Sep 10 20:58:32.881450 ip-10-0-128-236 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 20:58:33.299748 ip-10-0-128-236 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:58:33.299748 ip-10-0-128-236 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 20:58:33.299748 ip-10-0-128-236 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:58:33.299748 ip-10-0-128-236 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 20:58:33.299748 ip-10-0-128-236 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 20:58:33.301727 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.301641    2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 20:58:33.306546 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306530    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:33.306546 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306545    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306551    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306554    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306557    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306560    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306563    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306568    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306572    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306575    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306578    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306581    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306584    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306586    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306589    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306592    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306594    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306596    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306601    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306604    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:33.306617 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306607    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306610    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306613    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306615    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306618    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306620    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306623    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306627    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306631    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306634    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306637    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306639    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306642    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306644    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306648    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306651    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306653    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306656    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306658    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:33.307069 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306660    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306663    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306665    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306668    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306670    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306673    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306675    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306678    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306680    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306682    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306684    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306687    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306689    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306692    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306694    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306697    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306700    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306702    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306705    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:33.307586 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306707    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306710    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306713    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306716    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306718    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306720    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306723    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306726    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306728    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306731    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306734    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306736    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306739    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306742    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306744    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306747    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306749    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306751    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306754    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306757    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:33.308057 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306760    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306762    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306765    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306767    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306770    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306772    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306775    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306777    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306781    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306784    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306787    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306789    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306791    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306794    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306797    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306799    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306802    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306805    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306807    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306810    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:33.308567 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.306812    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308277    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308287    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308290    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308293    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308297    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308299    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308302    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308305    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308308    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308310    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308313    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308315    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308318    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308320    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308323    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308325    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308328    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308330    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308333    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:33.309253 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308337    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308340    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308342    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308345    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308348    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308350    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308353    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308355    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308357    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308361    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308365    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308368    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308370    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308373    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308375    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308378    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308381    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308385    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308388    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:33.310113 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308391    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308393    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308396    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308399    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308401    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308404    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308407    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308409    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308412    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308414    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308416    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308419    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308421    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308424    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308428    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308431    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308434    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308437    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308440    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:33.310765 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308442    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308452    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308455    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308458    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308460    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308463    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308465    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308468    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308470    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308473    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308476    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308478    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308481    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308483    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308486    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308488    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308491    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308493    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308497    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308499    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:33.311266 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308502    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308504    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308507    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308509    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308512    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308514    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308518    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308521    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308524    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308527    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308529    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308532    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308534    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308537    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308539    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308542    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308544    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308547    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308549    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308552    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:33.311758 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308554    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.308557    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310398    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310414    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310418    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310423    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310429    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310432    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310435    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310438    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310441    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310444    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310446    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310449    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310452    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310475    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310479    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310482    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310485    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:33.312275 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310488    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310491    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310493    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310496    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310499    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310501    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310504    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310507    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310509    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310512    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310514    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310517    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310519    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310523    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310525    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310528    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310531    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310536    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310541    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:33.312743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310545    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310550    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310553    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310556    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310559    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310562    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310565    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310567    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310570    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310572    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310575    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310578    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310580    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310583    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310586    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310589    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310591    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310594    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310596    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:33.313228 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310599    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310601    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310603    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310606    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310608    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310611    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310613    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310616    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310618    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310620    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310624    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310626    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310631    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310633    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310636    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310638    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310641    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310643    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310646    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310649    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:33.313684 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310651    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310654    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310656    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310659    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310662    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310664    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310667    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310669    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310678    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310681    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310683    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310686    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310688    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310691    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310694    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310696    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310699    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310701    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310704    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310706    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:33.314198 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310708    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310711    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310716    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.310719    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310800    2580 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310846    2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310854    2580 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310858    2580 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310862    2580 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310865    2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310870    2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310874    2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310877    2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310880    2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310883    2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310886    2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310889    2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310892    2580 flags.go:64] FLAG: --cgroup-root=""
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310896    2580 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310899    2580 flags.go:64] FLAG: --client-ca-file=""
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310901    2580 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310914    2580 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310921    2580 flags.go:64] FLAG: --cluster-domain=""
Sep 10 20:58:33.314669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310924    2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310928    2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310931    2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310934    2580 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310938    2580 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310941    2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310944    2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310947    2580 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310950    2580 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310952    2580 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310955    2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310960    2580 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310963    2580 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310968    2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310970    2580 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310973    2580 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310976    2580 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310979    2580 flags.go:64] FLAG: --enable-server="true"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310982    2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310992    2580 flags.go:64] FLAG: --event-burst="100"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310995    2580 flags.go:64] FLAG: --event-qps="50"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.310998    2580 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311001    2580 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311004    2580 flags.go:64] FLAG: --eviction-hard=""
Sep 10 20:58:33.315245 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311007    2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311010    2580 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311013    2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311016    2580 flags.go:64] FLAG: --eviction-soft=""
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311019    2580 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311021    2580 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311024    2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311033    2580 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311035    2580 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311038    2580 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311041    2580 flags.go:64] FLAG: --feature-gates=""
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311044    2580 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311047    2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311050    2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311053    2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311056    2580 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311059    2580 flags.go:64] FLAG: --help="false"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311062    2580 flags.go:64] FLAG: --hostname-override="ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311065    2580 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311069    2580 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311072    2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311075    2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311078    2580 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311081    2580 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 20:58:33.315822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311084    2580 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311087    2580 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311090    2580 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311092    2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311095    2580 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311099    2580 flags.go:64] FLAG: --kube-reserved=""
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311101    2580 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311104    2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311107    2580 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311110    2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311112    2580 flags.go:64] FLAG: --lock-file=""
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311115    2580 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311136    2580 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311140    2580 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311149    2580 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311152    2580 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311163    2580 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311166    2580 flags.go:64] FLAG: --logging-format="text"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311168    2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311172    2580 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311175    2580 flags.go:64] FLAG: --manifest-url=""
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311177    2580 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311182    2580 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311185    2580 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311189    2580 flags.go:64] FLAG: --max-pods="110"
Sep 10 20:58:33.316506 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311192    2580 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311195    2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311199    2580 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311201    2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311204    2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311207    2580 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311210    2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311218    2580 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311221    2580 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311224    2580 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311226    2580 flags.go:64] FLAG: --pod-cidr=""
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311229    2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311234    2580 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311237    2580 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311240    2580 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311243    2580 flags.go:64] FLAG: --port="10250"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311246    2580 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311248    2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0ce6008ff79c94c90"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311251    2580 flags.go:64] FLAG: --qos-reserved=""
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311254    2580 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311257    2580 flags.go:64] FLAG: --register-node="true"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311260    2580 flags.go:64] FLAG: --register-with-taints=""
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311263    2580 flags.go:64] FLAG: --registry-burst="10"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311266    2580 flags.go:64] FLAG: --registry-qps="5"
Sep 10 20:58:33.317173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311274    2580 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311277    2580 flags.go:64] FLAG: --reserved-memory=""
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311282    2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311285    2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311288    2580 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311290    2580 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311293    2580 flags.go:64] FLAG: --runonce="false"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311296    2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311299    2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311302    2580 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311305    2580 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311309    2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311312    2580 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311316    2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311319    2580 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311322    2580 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311324    2580 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311327    2580 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311330    2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311333    2580 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311336    2580 flags.go:64] FLAG: --system-cgroups=""
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311338    2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311344    2580 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311347    2580 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311350    2580 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 20:58:33.317764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311357    2580 flags.go:64] FLAG: --tls-min-version=""
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311359    2580 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311362    2580 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311365    2580 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311368    2580 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311371    2580 flags.go:64] FLAG: --v="2"
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311375    2580 flags.go:64] FLAG: --version="false"
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311378    2580 flags.go:64] FLAG: --vmodule=""
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311388    2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.311391    2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311518    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311522    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311525    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311528    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311533    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311536    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311539    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311544    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311548    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311551    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311553    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:33.318435 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311556    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311559    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311562    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311565    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311568    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311571    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311574    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311577    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311580    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311583    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311586    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311588    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311591    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311594    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311596    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311599    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311601    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311604    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311606    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311609    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:33.318973 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311618    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311620    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311623    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311626    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311628    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311631    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311634    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311636    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311640    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311644    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311648    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311651    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311654    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311656    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311659    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311662    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311664    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311667    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311669    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311672    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:33.319484 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311674    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311677    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311679    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311682    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311684    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311688    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311691    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311693    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311696    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311698    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311701    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311703    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311706    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311715    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311718    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311721    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311723    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311726    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311728    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311730    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:33.319971 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311735    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311739    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311741    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311744    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311747    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311749    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311752    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311754    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311757    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311760    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311763    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311766    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311768    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311771    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311774    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311777    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311779    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311782    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311784    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:33.320492 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311787    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311790    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311792    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311795    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311797    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311800    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311802    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311814    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.311817    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.312663    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.319785    2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 20:58:33.320952 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.319901    2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.321013    2580 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.321026    2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321088    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321093    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321097    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321100    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321103    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321106    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321108    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321111    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321114    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321128    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321130    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321133    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321136    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321138    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321141    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321143    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:33.321260 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321146    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321148    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321151    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321154    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321158    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321161    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321164    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321168    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321171    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321173    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321176    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321179    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321182    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321185    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321188    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321192    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321195    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321198    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321201    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:33.321712 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321204    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321208    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321212    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321214    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321218    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321220    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321223    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321225    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321228    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321230    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321233    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321236    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321239    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321241    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321244    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321246    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321249    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321252    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321254    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321257    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:33.322249 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321259    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321262    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321264    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321267    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321270    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321273    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321276    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321279    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321282    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321284    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321288    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321292    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321294    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321297    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321300    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321302    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321305    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321307    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321310    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321312    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:33.322744 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321315    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321318    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321320    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321323    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321325    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321328    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321331    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321333    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321336    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321338    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321340    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321343    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321346    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321348    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321351    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321353    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321356    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321358    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321361    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321363    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:33.323264 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321367    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321370    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321372    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321375    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.321380    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321487    2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321492    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321495    2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321498    2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321501    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321503    2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321506    2580 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321508    2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321511    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321514    2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 20:58:33.323743 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321517    2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321521    2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321524    2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321526    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321530    2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321534    2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321537    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321540    2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321542    2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321545    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321548    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321551    2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321554    2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321557    2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321560    2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321562    2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321565    2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321568    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321571    2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 20:58:33.324110 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321574    2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321577    2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321579    2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321582    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321584    2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321587    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321589    2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321592    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321594    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321597    2580 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321599    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321601    2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321604    2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321606    2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321609    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321611    2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321614    2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321616    2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321619    2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321621    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 20:58:33.324679 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321624    2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321626    2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321628    2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321631    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321633    2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321635    2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321638    2580 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321641    2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321643    2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321646    2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321648    2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321651    2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321654    2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321656    2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321659    2580 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321661    2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321663    2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321666    2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321669    2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321671    2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 20:58:33.325200 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321673    2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321676    2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321678    2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321681    2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321683    2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321686    2580 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321688    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321691    2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321694    2580 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321696    2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321700    2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321703    2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321706    2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321709    2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321712    2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321715    2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321718    2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321720    2580 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321723    2580 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321727    2580 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 20:58:33.325942 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321729    2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321732    2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321735    2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321737    2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321740    2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321743    2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321746    2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321748    2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321751    2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:33.321753    2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.321758    2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.321869    2580 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.324254    2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.325276    2580 server.go:1019] "Starting client certificate rotation"
Sep 10 20:58:33.326713 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.325389    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:58:33.327090 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.325444    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 20:58:33.327731 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.327717    2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:58:33.330407 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.330387    2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 20:58:33.342550 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.342532    2580 log.go:25] "Validated CRI v1 runtime API"
Sep 10 20:58:33.346421 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.346398    2580 log.go:25] "Validated CRI v1 image API"
Sep 10 20:58:33.347669 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.347652    2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 20:58:33.351020 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.350997    2580 fs.go:135] Filesystem UUIDs: map[35b4562c-e6ed-4cc1-a566-c0a846663590:/dev/nvme0n1p3 56f7958f-d358-41ff-88f1-af13deb8f051:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 10 20:58:33.351093 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.351018    2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 20:58:33.356247 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.356112    2580 manager.go:216] Machine: {Timestamp:2026-09-10 20:58:33.354632442 +0000 UTC m=+0.356878780 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099987 MemoryCapacity:32812150784 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28915b410197836fe52700c3377294 SystemUUID:ec28915b-4101-9783-6fe5-2700c3377294 BootID:888d5157-8398-4ca7-9e23-b47205078ee7 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7a:19:ac:12:83 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7a:19:ac:12:83 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:94:a8:65:05:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812150784 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 20:58:33.356247 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.356241    2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 20:58:33.356365 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.356329    2580 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 20:58:33.358938 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.358923    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:58:33.360031 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.360003    2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 20:58:33.360192 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.360034    2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 20:58:33.360796 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.360785    2580 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 20:58:33.360831 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.360799    2580 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 20:58:33.360831 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.360807    2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:58:33.361587 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.361577    2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 20:58:33.362688 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.362678    2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 20:58:33.366080 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366069    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:58:33.366216 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366206    2580 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 20:58:33.366252 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366242    2580 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 20:58:33.366280 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366253    2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 20:58:33.366280 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366267    2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 20:58:33.366280 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366276    2580 kubelet.go:398] "Adding apiserver pod source"
Sep 10 20:58:33.366380 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366284    2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 20:58:33.366629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366617    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:58:33.366674 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.366639    2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 20:58:33.369085 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.369068    2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 20:58:33.371982 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.371968    2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 20:58:33.372029 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.371999    2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 20:58:33.374256 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374241    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 20:58:33.374291 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374271    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 20:58:33.374291 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374283    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 20:58:33.374360 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374295    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 20:58:33.374360 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374311    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 20:58:33.374360 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374322    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 20:58:33.374360 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374347    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 20:58:33.374360 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374358    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 20:58:33.374483 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374370    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 20:58:33.374483 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374383    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 20:58:33.374483 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374410    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 20:58:33.374483 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.374419    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 20:58:33.376156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.376141    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 20:58:33.376189 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.376160    2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 20:58:33.376856 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.376837    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 20:58:33.376902 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.376877    2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 20:58:33.378198 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.378178    2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 20:58:33.379961 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.379949    2580 server.go:1268] "Started kubelet"
Sep 10 20:58:33.380060 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.380038    2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 20:58:33.380135 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.380099    2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 20:58:33.380193 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.380171    2580 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 20:58:33.381319 ip-10-0-128-236 systemd[1]: Started Kubernetes Kubelet.
Sep 10 20:58:33.381984 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.381968    2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 20:58:33.385629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.385604    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g7pm9"
Sep 10 20:58:33.386466 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.386430    2580 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 20:58:33.386691 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.386635    2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 20:58:33.390503 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.389363    2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-236.ec2.internal.18d4113ecbed92e1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-236.ec2.internal,UID:ip-10-0-128-236.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-236.ec2.internal,},FirstTimestamp:2026-09-10 20:58:33.379918561 +0000 UTC m=+0.382164899,LastTimestamp:2026-09-10 20:58:33.379918561 +0000 UTC m=+0.382164899,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-236.ec2.internal,}"
Sep 10 20:58:33.391587 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.391565    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g7pm9"
Sep 10 20:58:33.393507 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.393492    2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 20:58:33.393574 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.393531    2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 20:58:33.393641 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.393631    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:33.394275 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.394225    2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 20:58:33.394623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.394603    2580 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 20:58:33.394623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.394623    2580 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 20:58:33.394834 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.394784    2580 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 20:58:33.394834 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.394795    2580 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 20:58:33.394933 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.394916    2580 factory.go:153] Registering CRI-O factory
Sep 10 20:58:33.394982 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.394935    2580 factory.go:223] Registration of the crio container factory successfully
Sep 10 20:58:33.395028 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.395005    2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 20:58:33.395028 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.395015    2580 factory.go:55] Registering systemd factory
Sep 10 20:58:33.395028 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.395024    2580 factory.go:223] Registration of the systemd container factory successfully
Sep 10 20:58:33.395155 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.395041    2580 factory.go:103] Registering Raw factory
Sep 10 20:58:33.395155 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.395055    2580 manager.go:1196] Started watching for new ooms in manager
Sep 10 20:58:33.395579 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.395561    2580 manager.go:349] Starting recovery of all containers
Sep 10 20:58:33.398466 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.398440    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:33.401743 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.401719    2580 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-236.ec2.internal\" not found" node="ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.407436 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.407418    2580 manager.go:354] Recovery completed
Sep 10 20:58:33.411756 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.411743    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:33.413564 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.413549    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:33.413633 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.413581    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:33.413633 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.413592    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:33.414221 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.414199    2580 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 20:58:33.414221 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.414212    2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 20:58:33.414221 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.414214    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:33.414391 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.414228    2580 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 20:58:33.414966 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.414949    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:33.415036 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.414982    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:33.415036 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.414994    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:33.415574 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.415562    2580 policy_none.go:49] "None policy: Start"
Sep 10 20:58:33.415618 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.415579    2580 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 20:58:33.415618 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.415589    2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 20:58:33.416667 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.416656    2580 policy_none.go:47] "Start"
Sep 10 20:58:33.457858 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.457841    2580 manager.go:337] "Starting Device Plugin manager"
Sep 10 20:58:33.457939 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.457899    2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 20:58:33.457939 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.457908    2580 server.go:87] "Starting device plugin registration server"
Sep 10 20:58:33.458283 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.458094    2580 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 20:58:33.458283 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.458105    2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 20:58:33.458396 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.458361    2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 20:58:33.459049 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.458434    2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 20:58:33.459049 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.458447    2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 20:58:33.459363 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.459326    2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 20:58:33.474760 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.459370    2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:33.511134 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.511094    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 20:58:33.512392 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.512372    2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 20:58:33.512486 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.512418    2580 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 20:58:33.512486 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.512438    2580 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 20:58:33.512594 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.512487    2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 20:58:33.514902 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.514882    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:33.558767 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.558719    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:33.559677 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.559658    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:33.559770 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.559696    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:33.559770 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.559711    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:33.559770 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.559740    2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.568069 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.568054    2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.568148 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.568075    2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-236.ec2.internal\": node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:33.577966 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.577946    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:33.612570 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.612548    2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"]
Sep 10 20:58:33.612628 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.612608    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:33.613988 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.613968    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:33.614039 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.613996    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:33.614039 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.614007    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:33.615332 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.615320    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:33.615471 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.615457    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.615508 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.615485    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:33.616077 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.616063    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:33.616157 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.616086    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:33.616157 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.616098    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:33.616157 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.616063    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:33.616157 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.616156    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:33.616293 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.616167    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:33.617720 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.617706    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.617792 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.617730    2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 20:58:33.618372 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.618354    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 20:58:33.618372 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.618383    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 20:58:33.618507 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.618393    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeHasSufficientPID"
Sep 10 20:58:33.641626 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.641607    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-236.ec2.internal\" not found" node="ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.645800 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.645781    2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-236.ec2.internal\" not found" node="ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.678850 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.678825    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:33.697470 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.697440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23dc3affb2184a97e7389ef44e84c8b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"23dc3affb2184a97e7389ef44e84c8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.697557 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.697474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23dc3affb2184a97e7389ef44e84c8b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"23dc3affb2184a97e7389ef44e84c8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.697557 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.697495    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc4b39e75aad3eda53248a12852cd31a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"dc4b39e75aad3eda53248a12852cd31a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.779509 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.779475    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:33.797800 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.797778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23dc3affb2184a97e7389ef44e84c8b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"23dc3affb2184a97e7389ef44e84c8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.797859 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.797806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23dc3affb2184a97e7389ef44e84c8b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"23dc3affb2184a97e7389ef44e84c8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.797859 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.797827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc4b39e75aad3eda53248a12852cd31a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"dc4b39e75aad3eda53248a12852cd31a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.797930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.797872    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/dc4b39e75aad3eda53248a12852cd31a-config\") pod \"kube-apiserver-proxy-ip-10-0-128-236.ec2.internal\" (UID: \"dc4b39e75aad3eda53248a12852cd31a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.797930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.797880    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/23dc3affb2184a97e7389ef44e84c8b0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"23dc3affb2184a97e7389ef44e84c8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.797930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.797890    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/23dc3affb2184a97e7389ef44e84c8b0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal\" (UID: \"23dc3affb2184a97e7389ef44e84c8b0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.880240 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.880172    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:33.944630 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.944607    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.949183 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:33.948906    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:33.980636 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:33.980602    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:34.081095 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:34.081066    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:34.181658 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:34.181587    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:34.282181 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:34.282150    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:34.325604 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.325578    2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 20:58:34.326201 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.325703    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:58:34.326201 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.325735    2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 20:58:34.382499 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:34.382477    2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-236.ec2.internal\" not found"
Sep 10 20:58:34.393017 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.392990    2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:34.394476 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.394451    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:34.394542 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.394475    2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 20:58:34.394542 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.394480    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 20:53:33 +0000 UTC" deadline="2028-02-14 13:00:09.375723118 +0000 UTC"
Sep 10 20:58:34.394542 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.394519    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12520h1m34.981208474s"
Sep 10 20:58:34.407180 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.407164    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:58:34.408930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.408915    2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal"
Sep 10 20:58:34.409291 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.409276    2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 20:58:34.417492 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.417470    2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 20:58:34.429997 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.429977    2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nmqzq"
Sep 10 20:58:34.441427 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.441365    2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nmqzq"
Sep 10 20:58:34.537138 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:34.537096    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4b39e75aad3eda53248a12852cd31a.slice/crio-bba67aefc78e9321d6eab398603d48d58aef81d03d2ec59c72e3792b5823347a WatchSource:0}: Error finding container bba67aefc78e9321d6eab398603d48d58aef81d03d2ec59c72e3792b5823347a: Status 404 returned error can't find the container with id bba67aefc78e9321d6eab398603d48d58aef81d03d2ec59c72e3792b5823347a
Sep 10 20:58:34.537331 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:34.537313    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23dc3affb2184a97e7389ef44e84c8b0.slice/crio-99c8da3a1c97c08095956e636a6a92541f1bd528a94113f9613d61970ba87613 WatchSource:0}: Error finding container 99c8da3a1c97c08095956e636a6a92541f1bd528a94113f9613d61970ba87613: Status 404 returned error can't find the container with id 99c8da3a1c97c08095956e636a6a92541f1bd528a94113f9613d61970ba87613
Sep 10 20:58:34.541735 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.541712    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 20:58:34.650248 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:34.650062    2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:35.367327 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.367280    2580 apiserver.go:52] "Watching apiserver"
Sep 10 20:58:35.376596 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.376561    2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 20:58:35.377795 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.377760    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-xcgkb","openshift-cluster-node-tuning-operator/tuned-lfjp5","openshift-dns/node-resolver-gp8hn","openshift-multus/network-metrics-daemon-8wqmx","openshift-network-diagnostics/network-check-target-kd8gq","openshift-network-operator/iptables-alerter-lnz7h","openshift-ovn-kubernetes/ovnkube-node-2dnpx","kube-system/global-pull-secret-syncer-ks95v","kube-system/konnectivity-agent-wq6lz","kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj","openshift-image-registry/node-ca-ctnsl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal","openshift-multus/multus-additional-cni-plugins-pcz7g"]
Sep 10 20:58:35.379685 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.379661    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.381248 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.381227    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.382305 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.382282    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 20:58:35.382430 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.382344    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 20:58:35.382564 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.382548    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 20:58:35.382639 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.382585    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 20:58:35.383230 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.382761    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 20:58:35.383230 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.382841    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 20:58:35.383230 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.382846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.383230 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.382976    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2sptm\""
Sep 10 20:58:35.383230 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.383089    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 20:58:35.384391 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.384295    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ddn26\""
Sep 10 20:58:35.384619 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.384598    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 20:58:35.384999 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.384975    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 20:58:35.385136 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.385114    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 20:58:35.385408 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.385386    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d7mp8\""
Sep 10 20:58:35.385507 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.385411    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 20:58:35.385659 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.385640    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:35.385763 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.385732    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:35.386388 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.386078    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:35.386388 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.386147    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:35.387629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.387604    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.389190 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.389166    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:35.389759 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.389740    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:58:35.389849 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.389820    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4gzfv\""
Sep 10 20:58:35.390061 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.390033    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 20:58:35.390118 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.390063    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 20:58:35.391157 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.391139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.391735 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.391710    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 20:58:35.391807 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.391784    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 20:58:35.391940 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.391924    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cfp2h\""
Sep 10 20:58:35.392740 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.392717    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.393249 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.393232    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 20:58:35.393547 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.393531    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 20:58:35.393964 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.393729    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 20:58:35.393964 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.393802    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 20:58:35.393964 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.393857    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 20:58:35.394406 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.394387    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6cthj\""
Sep 10 20:58:35.395244 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.395213    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.395244 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.395228    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:58:35.395244 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.395243    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 20:58:35.395484 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.395288    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:35.395484 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.395341    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xslb9\""
Sep 10 20:58:35.396534 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.396510    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.397871 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.397852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.398847 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.398827    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-pzvmp\""
Sep 10 20:58:35.398915 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.398869    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 20:58:35.401241 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.401220    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 20:58:35.401350 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.401326    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qkcgh\""
Sep 10 20:58:35.401424 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.401328    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 20:58:35.401424 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.401383    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 20:58:35.404974 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.404947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-run\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.404974 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.404970    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-host\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.404998    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-run-netns\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405028    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-node-log\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/717d024e-fb07-41d1-b9d2-f2180c6bad68-host-slash\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405062    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-kubelet\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-var-lib-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjshs\" (UniqueName: \"kubernetes.io/projected/717d024e-fb07-41d1-b9d2-f2180c6bad68-kube-api-access-hjshs\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405133    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0cf2fb43-34ef-4996-b746-31482ad247ab-kubelet-config\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.405156 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405150    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405166    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-slash\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405182    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-hosts-file\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/717d024e-fb07-41d1-b9d2-f2180c6bad68-iptables-alerter-script\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405235    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-tuned\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405251    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405268    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405286    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72x8j\" (UniqueName: \"kubernetes.io/projected/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-kube-api-access-72x8j\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405309    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7e62b8-d8b2-431e-8d20-27577cd0f03c-agent-certs\") pod \"konnectivity-agent-wq6lz\" (UID: \"0b7e62b8-d8b2-431e-8d20-27577cd0f03c\") " pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405347    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-systemd\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405364    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405381    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysconfig\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405398    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-systemd\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405414    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0cf2fb43-34ef-4996-b746-31482ad247ab-dbus\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-socket-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.405623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-run-ovn-kubernetes\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405469    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405487    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-env-overrides\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405503    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-sys\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/857fc54f-2c08-4613-84d4-40a4c8b3152a-tmp\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405539    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqljl\" (UniqueName: \"kubernetes.io/projected/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-kube-api-access-xqljl\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7e62b8-d8b2-431e-8d20-27577cd0f03c-konnectivity-ca\") pod \"konnectivity-agent-wq6lz\" (UID: \"0b7e62b8-d8b2-431e-8d20-27577cd0f03c\") " pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405580    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cnibin\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405601    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-etc-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405636    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovn-node-metrics-cert\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405653    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-lib-modules\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-sys-fs\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405686    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovnkube-config\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-kubernetes\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysctl-conf\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405737    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-var-lib-kubelet\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.406738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405761    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv8c8\" (UniqueName: \"kubernetes.io/projected/857fc54f-2c08-4613-84d4-40a4c8b3152a-kube-api-access-sv8c8\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405795    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzkk6\" (UniqueName: \"kubernetes.io/projected/9341a00d-f1e0-4798-b687-8f058c4e2d59-kube-api-access-bzkk6\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-systemd-units\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405827    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-log-socket\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-cni-netd\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405860    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t94h\" (UniqueName: \"kubernetes.io/projected/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-kube-api-access-4t94h\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405877    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405900    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-modprobe-d\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405916    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-registration-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405935    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-cni-bin\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.405991    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-device-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.406007    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-system-cni-dir\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.406027    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjgzj\" (UniqueName: \"kubernetes.io/projected/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-kube-api-access-qjgzj\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.407629 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.406043    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-ovn\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.408423 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.406060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovnkube-script-lib\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.408423 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.406078    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-tmp-dir\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.408423 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.406095    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysctl-d\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.426437 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.426416    2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:35.443695 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.443644    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:53:34 +0000 UTC" deadline="2028-06-11 20:09:34.313566928 +0000 UTC"
Sep 10 20:58:35.443839 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.443696    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15359h10m58.869875639s"
Sep 10 20:58:35.495103 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.495071    2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 20:58:35.496071 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.496049    2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 20:58:35.506590 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506562    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.506704 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506607    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjshs\" (UniqueName: \"kubernetes.io/projected/717d024e-fb07-41d1-b9d2-f2180c6bad68-kube-api-access-hjshs\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.506704 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506633    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0cf2fb43-34ef-4996-b746-31482ad247ab-kubelet-config\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.506704 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506651    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.506704 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.506930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506718    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptf7r\" (UniqueName: \"kubernetes.io/projected/4a77b4af-edcb-478c-8d57-4b7b29339a22-kube-api-access-ptf7r\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.506930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqj9g\" (UniqueName: \"kubernetes.io/projected/372eb974-c525-4f33-954a-23c699ac2016-kube-api-access-zqj9g\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.506930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-slash\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.506930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506805    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-hosts-file\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.506930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/717d024e-fb07-41d1-b9d2-f2180c6bad68-iptables-alerter-script\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.506930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-tuned\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.506930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.506930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506915    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72x8j\" (UniqueName: \"kubernetes.io/projected/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-kube-api-access-72x8j\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506961    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.506977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7e62b8-d8b2-431e-8d20-27577cd0f03c-agent-certs\") pod \"konnectivity-agent-wq6lz\" (UID: \"0b7e62b8-d8b2-431e-8d20-27577cd0f03c\") " pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507000    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-systemd\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507061    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-cnibin\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0cf2fb43-34ef-4996-b746-31482ad247ab-kubelet-config\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507084    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a77b4af-edcb-478c-8d57-4b7b29339a22-cni-binary-copy\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507123    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-cni-multus\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507140    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysconfig\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507156    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-systemd\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507199    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0cf2fb43-34ef-4996-b746-31482ad247ab-dbus\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-socket-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507250    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-system-cni-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-k8s-cni-cncf-io\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507281    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cni-binary-copy\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.507366 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507320    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-run-ovn-kubernetes\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507365    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507410    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-env-overrides\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-sys\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507472    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/857fc54f-2c08-4613-84d4-40a4c8b3152a-tmp\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqljl\" (UniqueName: \"kubernetes.io/projected/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-kube-api-access-xqljl\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507506    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7e62b8-d8b2-431e-8d20-27577cd0f03c-konnectivity-ca\") pod \"konnectivity-agent-wq6lz\" (UID: \"0b7e62b8-d8b2-431e-8d20-27577cd0f03c\") " pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cnibin\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507570    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-etc-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507601    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovn-node-metrics-cert\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507626    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-lib-modules\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507624    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-sys-fs\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507701    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-hosts-file\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507716    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovnkube-config\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-kubernetes\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.508250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507765    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysctl-conf\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-var-lib-kubelet\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507834    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv8c8\" (UniqueName: \"kubernetes.io/projected/857fc54f-2c08-4613-84d4-40a4c8b3152a-kube-api-access-sv8c8\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507895    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzkk6\" (UniqueName: \"kubernetes.io/projected/9341a00d-f1e0-4798-b687-8f058c4e2d59-kube-api-access-bzkk6\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507922    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-cni-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-systemd-units\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507954    2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507966    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-log-socket\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-cni-netd\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t94h\" (UniqueName: \"kubernetes.io/projected/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-kube-api-access-4t94h\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508095    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-slash\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508119    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-cni-netd\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508229    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-socket-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508231    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-run-ovn-kubernetes\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508298    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-sys-fs\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508341    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-kubernetes\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.509148 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.508402    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.508481    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs podName:9341a00d-f1e0-4798-b687-8f058c4e2d59 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:36.008443688 +0000 UTC m=+3.010690016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs") pod "network-metrics-daemon-8wqmx" (UID: "9341a00d-f1e0-4798-b687-8f058c4e2d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508699    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0b7e62b8-d8b2-431e-8d20-27577cd0f03c-konnectivity-ca\") pod \"konnectivity-agent-wq6lz\" (UID: \"0b7e62b8-d8b2-431e-8d20-27577cd0f03c\") " pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.508780    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-env-overrides\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.507975    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-systemd\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509074    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-sys\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509117    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-cnibin\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-etc-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509157    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovnkube-config\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509259    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-modprobe-d\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509288    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-registration-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509319    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-socket-dir-parent\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509353    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509378    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/717d024e-fb07-41d1-b9d2-f2180c6bad68-iptables-alerter-script\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509391    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-hostroot\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.510045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509417    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-conf-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509453    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysconfig\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-etc-kubernetes\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-cni-bin\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509503    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-systemd\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509521    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509643    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0cf2fb43-34ef-4996-b746-31482ad247ab-dbus\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-lib-modules\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509763    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-device-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-system-cni-dir\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjgzj\" (UniqueName: \"kubernetes.io/projected/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-kube-api-access-qjgzj\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509833    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/372eb974-c525-4f33-954a-23c699ac2016-serviceca\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-ovn\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovnkube-script-lib\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-device-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-tmp-dir\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.510930 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysctl-conf\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.509967    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysctl-d\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.509985    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510063    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-system-cni-dir\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.510078    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret podName:0cf2fb43-34ef-4996-b746-31482ad247ab nodeName:}" failed. No retries permitted until 2026-09-10 20:58:36.010059785 +0000 UTC m=+3.012306119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret") pod "global-pull-secret-syncer-ks95v" (UID: "0cf2fb43-34ef-4996-b746-31482ad247ab") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510131    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-var-lib-kubelet\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-etc-selinux\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-tmp-dir\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-run\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510413    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-host\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510444    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-netns\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510502    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-systemd-units\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510505    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510512    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-modprobe-d\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510610    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-sysctl-d\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510817    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-kubelet\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510854    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-run-netns\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.511750 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510883    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-node-log\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-registration-dir\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510914    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/717d024e-fb07-41d1-b9d2-f2180c6bad68-host-slash\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-daemon-config\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.510954    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-cni-bin\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511002    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/372eb974-c525-4f33-954a-23c699ac2016-host\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-run-netns\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511035    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-kubelet\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-node-log\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-cni-bin\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511097    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-multus-certs\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511125    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/717d024e-fb07-41d1-b9d2-f2180c6bad68-host-slash\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511128    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-var-lib-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511186    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-run-ovn\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-var-lib-openvswitch\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511239    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-host-kubelet\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-run\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/857fc54f-2c08-4613-84d4-40a4c8b3152a-host\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.512384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovnkube-script-lib\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.513130 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511593    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-log-socket\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.513130 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.511948    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0b7e62b8-d8b2-431e-8d20-27577cd0f03c-agent-certs\") pod \"konnectivity-agent-wq6lz\" (UID: \"0b7e62b8-d8b2-431e-8d20-27577cd0f03c\") " pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:35.514237 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.514211    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-ovn-node-metrics-cert\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.514565 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.514547    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/857fc54f-2c08-4613-84d4-40a4c8b3152a-etc-tuned\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.515989 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.515963    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjshs\" (UniqueName: \"kubernetes.io/projected/717d024e-fb07-41d1-b9d2-f2180c6bad68-kube-api-access-hjshs\") pod \"iptables-alerter-lnz7h\" (UID: \"717d024e-fb07-41d1-b9d2-f2180c6bad68\") " pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.516830 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.516066    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/857fc54f-2c08-4613-84d4-40a4c8b3152a-tmp\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.516830 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.516617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72x8j\" (UniqueName: \"kubernetes.io/projected/ad2e3059-c7d0-405d-8ff2-8a97f0644f81-kube-api-access-72x8j\") pod \"ovnkube-node-2dnpx\" (UID: \"ad2e3059-c7d0-405d-8ff2-8a97f0644f81\") " pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.516985 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.516774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqljl\" (UniqueName: \"kubernetes.io/projected/10564b7c-e01c-4287-9ecf-f01ebfbfe70c-kube-api-access-xqljl\") pod \"aws-ebs-csi-driver-node-fwqhj\" (UID: \"10564b7c-e01c-4287-9ecf-f01ebfbfe70c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.517546 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.517505    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" event={"ID":"dc4b39e75aad3eda53248a12852cd31a","Type":"ContainerStarted","Data":"bba67aefc78e9321d6eab398603d48d58aef81d03d2ec59c72e3792b5823347a"}
Sep 10 20:58:35.518905 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.518837    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"23dc3affb2184a97e7389ef44e84c8b0","Type":"ContainerStarted","Data":"99c8da3a1c97c08095956e636a6a92541f1bd528a94113f9613d61970ba87613"}
Sep 10 20:58:35.519975 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.519944    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:35.519975 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.519971    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:35.520157 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.519985    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s9jqs for pod openshift-network-diagnostics/network-check-target-kd8gq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:35.520157 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:35.520041    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs podName:784ee7bb-d459-4694-b60e-10ef820b238a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:36.020024732 +0000 UTC m=+3.022271081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-s9jqs" (UniqueName: "kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs") pod "network-check-target-kd8gq" (UID: "784ee7bb-d459-4694-b60e-10ef820b238a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:35.521695 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.521670    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjgzj\" (UniqueName: \"kubernetes.io/projected/1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25-kube-api-access-qjgzj\") pod \"multus-additional-cni-plugins-pcz7g\" (UID: \"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25\") " pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.522459 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.522430    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t94h\" (UniqueName: \"kubernetes.io/projected/7ff24c61-0ba5-4a58-a41c-95284c27b8d3-kube-api-access-4t94h\") pod \"node-resolver-gp8hn\" (UID: \"7ff24c61-0ba5-4a58-a41c-95284c27b8d3\") " pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.526117 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.523679    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzkk6\" (UniqueName: \"kubernetes.io/projected/9341a00d-f1e0-4798-b687-8f058c4e2d59-kube-api-access-bzkk6\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:35.526117 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.525892    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv8c8\" (UniqueName: \"kubernetes.io/projected/857fc54f-2c08-4613-84d4-40a4c8b3152a-kube-api-access-sv8c8\") pod \"tuned-lfjp5\" (UID: \"857fc54f-2c08-4613-84d4-40a4c8b3152a\") " pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.612089 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612053    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-system-cni-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612276 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612106    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-k8s-cni-cncf-io\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612276 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612152    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-system-cni-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612276 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612168    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-cni-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612276 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612222    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-k8s-cni-cncf-io\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612276 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612227    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-socket-dir-parent\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-hostroot\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612293    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-socket-dir-parent\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612246    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-cni-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612307    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-conf-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-etc-kubernetes\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-conf-dir\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/372eb974-c525-4f33-954a-23c699ac2016-serviceca\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612381    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-etc-kubernetes\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612384    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-netns\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-kubelet\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-hostroot\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612425    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-daemon-config\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612459    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/372eb974-c525-4f33-954a-23c699ac2016-host\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-cni-bin\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612519    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-multus-certs\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/372eb974-c525-4f33-954a-23c699ac2016-host\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.612552 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612557    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptf7r\" (UniqueName: \"kubernetes.io/projected/4a77b4af-edcb-478c-8d57-4b7b29339a22-kube-api-access-ptf7r\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612556    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-netns\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqj9g\" (UniqueName: \"kubernetes.io/projected/372eb974-c525-4f33-954a-23c699ac2016-kube-api-access-zqj9g\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-cni-bin\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612617    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-run-multus-certs\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612632    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-cnibin\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612676    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a77b4af-edcb-478c-8d57-4b7b29339a22-cni-binary-copy\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-cnibin\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612705    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-cni-multus\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612766    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-cni-multus\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612769    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/372eb974-c525-4f33-954a-23c699ac2016-serviceca\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612568    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4a77b4af-edcb-478c-8d57-4b7b29339a22-host-var-lib-kubelet\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.612962    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4a77b4af-edcb-478c-8d57-4b7b29339a22-multus-daemon-config\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.613227 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.613143    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4a77b4af-edcb-478c-8d57-4b7b29339a22-cni-binary-copy\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.624690 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.624607    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqj9g\" (UniqueName: \"kubernetes.io/projected/372eb974-c525-4f33-954a-23c699ac2016-kube-api-access-zqj9g\") pod \"node-ca-ctnsl\" (UID: \"372eb974-c525-4f33-954a-23c699ac2016\") " pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:35.624807 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.624692    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptf7r\" (UniqueName: \"kubernetes.io/projected/4a77b4af-edcb-478c-8d57-4b7b29339a22-kube-api-access-ptf7r\") pod \"multus-xcgkb\" (UID: \"4a77b4af-edcb-478c-8d57-4b7b29339a22\") " pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.702893 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.702856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:58:35.713384 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.713352    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj"
Sep 10 20:58:35.721181 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.721157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gp8hn"
Sep 10 20:58:35.726657 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.726632    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lnz7h"
Sep 10 20:58:35.733429 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.733410    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:35.741356 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.741338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pcz7g"
Sep 10 20:58:35.747082 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.747063    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lfjp5"
Sep 10 20:58:35.754032 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.754014    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xcgkb"
Sep 10 20:58:35.759766 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:35.759737    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctnsl"
Sep 10 20:58:36.016421 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:36.016337    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:36.016421 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:36.016392    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:36.016614 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.016501    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:36.016614 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.016515    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:36.016614 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.016559    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs podName:9341a00d-f1e0-4798-b687-8f058c4e2d59 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:37.016540735 +0000 UTC m=+4.018787067 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs") pod "network-metrics-daemon-8wqmx" (UID: "9341a00d-f1e0-4798-b687-8f058c4e2d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:36.016614 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.016584    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret podName:0cf2fb43-34ef-4996-b746-31482ad247ab nodeName:}" failed. No retries permitted until 2026-09-10 20:58:37.016564245 +0000 UTC m=+4.018810583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret") pod "global-pull-secret-syncer-ks95v" (UID: "0cf2fb43-34ef-4996-b746-31482ad247ab") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:36.117757 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:36.117700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:36.117926 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.117903    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:36.117999 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.117932    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:36.117999 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.117945    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s9jqs for pod openshift-network-diagnostics/network-check-target-kd8gq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:36.118113 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.118015    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs podName:784ee7bb-d459-4694-b60e-10ef820b238a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:37.117992237 +0000 UTC m=+4.120238589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9jqs" (UniqueName: "kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs") pod "network-check-target-kd8gq" (UID: "784ee7bb-d459-4694-b60e-10ef820b238a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:36.444832 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:36.444772    2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 20:53:34 +0000 UTC" deadline="2028-03-16 19:03:45.694204152 +0000 UTC"
Sep 10 20:58:36.444832 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:36.444823    2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13270h5m9.249383882s"
Sep 10 20:58:36.513633 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:36.513608    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:36.513730 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.513713    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:36.513778 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:36.513749    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:36.513813 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:36.513800    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:36.519080 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:36.519039    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a77b4af_edcb_478c_8d57_4b7b29339a22.slice/crio-3efc4e8642553be3cda1134183cfbc068081c22f6725e98ed22c0a78d5859d6d WatchSource:0}: Error finding container 3efc4e8642553be3cda1134183cfbc068081c22f6725e98ed22c0a78d5859d6d: Status 404 returned error can't find the container with id 3efc4e8642553be3cda1134183cfbc068081c22f6725e98ed22c0a78d5859d6d
Sep 10 20:58:36.520500 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:36.520474    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10564b7c_e01c_4287_9ecf_f01ebfbfe70c.slice/crio-fe46cf47d3409c6a4b21b8d381fab1ec9673b45cf9c97bba34ec0bf2c2b3ca2c WatchSource:0}: Error finding container fe46cf47d3409c6a4b21b8d381fab1ec9673b45cf9c97bba34ec0bf2c2b3ca2c: Status 404 returned error can't find the container with id fe46cf47d3409c6a4b21b8d381fab1ec9673b45cf9c97bba34ec0bf2c2b3ca2c
Sep 10 20:58:36.523812 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:58:36.523461    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2e3059_c7d0_405d_8ff2_8a97f0644f81.slice/crio-7c2be98b7556bcf3feb4837c42b507a1931fd70fb18e1241e43a7a038eb98749 WatchSource:0}: Error finding container 7c2be98b7556bcf3feb4837c42b507a1931fd70fb18e1241e43a7a038eb98749: Status 404 returned error can't find the container with id 7c2be98b7556bcf3feb4837c42b507a1931fd70fb18e1241e43a7a038eb98749
Sep 10 20:58:37.023233 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.023127    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:37.023233 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.023215    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:37.023424 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.023390    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:37.023470 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.023455    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs podName:9341a00d-f1e0-4798-b687-8f058c4e2d59 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:39.023435169 +0000 UTC m=+6.025681515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs") pod "network-metrics-daemon-8wqmx" (UID: "9341a00d-f1e0-4798-b687-8f058c4e2d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:37.023955 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.023863    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:37.023955 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.023918    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret podName:0cf2fb43-34ef-4996-b746-31482ad247ab nodeName:}" failed. No retries permitted until 2026-09-10 20:58:39.023902617 +0000 UTC m=+6.026148947 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret") pod "global-pull-secret-syncer-ks95v" (UID: "0cf2fb43-34ef-4996-b746-31482ad247ab") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:37.124806 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.124119    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:37.124806 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.124323    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:37.124806 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.124349    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:37.124806 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.124362    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s9jqs for pod openshift-network-diagnostics/network-check-target-kd8gq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:37.124806 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.124428    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs podName:784ee7bb-d459-4694-b60e-10ef820b238a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:39.124407637 +0000 UTC m=+6.126653984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9jqs" (UniqueName: "kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs") pod "network-check-target-kd8gq" (UID: "784ee7bb-d459-4694-b60e-10ef820b238a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:37.526186 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.526152    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:37.526659 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:37.526320    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:37.539125 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.539091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerStarted","Data":"1c1576ef291c0f7ba3c0f08b72d68beb564c5df82e1ea05290d7e3165925346f"}
Sep 10 20:58:37.549997 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.549959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gp8hn" event={"ID":"7ff24c61-0ba5-4a58-a41c-95284c27b8d3","Type":"ContainerStarted","Data":"1aedcd1e209c7dd41b1a4145683a90543b503a8f00842af13c93f17ae96fba73"}
Sep 10 20:58:37.562309 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.561996    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wq6lz" event={"ID":"0b7e62b8-d8b2-431e-8d20-27577cd0f03c","Type":"ContainerStarted","Data":"7433e62d87f5d9e1dfc4765dc9191369c9be7ce205ea967c8c231f0022c3be7c"}
Sep 10 20:58:37.585718 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.585616    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctnsl" event={"ID":"372eb974-c525-4f33-954a-23c699ac2016","Type":"ContainerStarted","Data":"5ef1ce10a85412065e073bc3b1ba7bd91c689d5680b6f43a35c9c6736dc523fd"}
Sep 10 20:58:37.598528 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.598396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"7c2be98b7556bcf3feb4837c42b507a1931fd70fb18e1241e43a7a038eb98749"}
Sep 10 20:58:37.605143 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.605047    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj" event={"ID":"10564b7c-e01c-4287-9ecf-f01ebfbfe70c","Type":"ContainerStarted","Data":"fe46cf47d3409c6a4b21b8d381fab1ec9673b45cf9c97bba34ec0bf2c2b3ca2c"}
Sep 10 20:58:37.615678 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.615648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lnz7h" event={"ID":"717d024e-fb07-41d1-b9d2-f2180c6bad68","Type":"ContainerStarted","Data":"6a65fc42ebabc1e241298ea6f67cddf6113f86ac3ade929391354bed5021aee8"}
Sep 10 20:58:37.626906 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.626828    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lfjp5" event={"ID":"857fc54f-2c08-4613-84d4-40a4c8b3152a","Type":"ContainerStarted","Data":"de6be4f702ae8b860ccafb7fc6fba2948062dd3a7e134a252e629973e2ff5a1c"}
Sep 10 20:58:37.637090 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.637004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xcgkb" event={"ID":"4a77b4af-edcb-478c-8d57-4b7b29339a22","Type":"ContainerStarted","Data":"3efc4e8642553be3cda1134183cfbc068081c22f6725e98ed22c0a78d5859d6d"}
Sep 10 20:58:37.658485 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.658251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" event={"ID":"dc4b39e75aad3eda53248a12852cd31a","Type":"ContainerStarted","Data":"5972dc1b2cfd99f805cf271b43bef9a2d6b3e2250639d61dbd92e4bc5339ee0d"}
Sep 10 20:58:37.664966 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.664428    2580 generic.go:356] "Generic (PLEG): container finished" podID="23dc3affb2184a97e7389ef44e84c8b0" containerID="cf2ce2107f6618c9481244660fabb3e5a5a1af910ac92b338584ec5b1adc5153" exitCode=0
Sep 10 20:58:37.664966 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.664466    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"23dc3affb2184a97e7389ef44e84c8b0","Type":"ContainerDied","Data":"cf2ce2107f6618c9481244660fabb3e5a5a1af910ac92b338584ec5b1adc5153"}
Sep 10 20:58:37.692424 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:37.692370    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-236.ec2.internal" podStartSLOduration=3.692349123 podStartE2EDuration="3.692349123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:37.675177838 +0000 UTC m=+4.677424190" watchObservedRunningTime="2026-09-10 20:58:37.692349123 +0000 UTC m=+4.694595479"
Sep 10 20:58:38.513521 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:38.513471    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:38.513709 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:38.513602    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:38.513993 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:38.513973    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:38.514163 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:38.514072    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:38.684259 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:38.684222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" event={"ID":"23dc3affb2184a97e7389ef44e84c8b0","Type":"ContainerStarted","Data":"15ba169ce816de344b5680c8548970b917f53117f134ac6495bcba555da9f62b"}
Sep 10 20:58:39.042549 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:39.042508    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:39.042743 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:39.042583    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:39.042803 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.042746    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:39.042851 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.042812    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs podName:9341a00d-f1e0-4798-b687-8f058c4e2d59 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:43.042794305 +0000 UTC m=+10.045040633 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs") pod "network-metrics-daemon-8wqmx" (UID: "9341a00d-f1e0-4798-b687-8f058c4e2d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:39.043232 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.043212    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:39.043331 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.043265    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret podName:0cf2fb43-34ef-4996-b746-31482ad247ab nodeName:}" failed. No retries permitted until 2026-09-10 20:58:43.043250437 +0000 UTC m=+10.045496778 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret") pod "global-pull-secret-syncer-ks95v" (UID: "0cf2fb43-34ef-4996-b746-31482ad247ab") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:39.142997 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:39.142960    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:39.143191 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.143149    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:39.143191 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.143168    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:39.143191 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.143179    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s9jqs for pod openshift-network-diagnostics/network-check-target-kd8gq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:39.143346 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.143248    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs podName:784ee7bb-d459-4694-b60e-10ef820b238a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:43.143229469 +0000 UTC m=+10.145475818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9jqs" (UniqueName: "kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs") pod "network-check-target-kd8gq" (UID: "784ee7bb-d459-4694-b60e-10ef820b238a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:39.512982 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:39.512678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:39.524243 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:39.521332    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:40.513225 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:40.512600    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:40.513225 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:40.512718    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:40.513225 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:40.513084    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:40.513225 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:40.513185    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:41.514757 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:41.514374    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:41.514757 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:41.514487    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:42.514320 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:42.513681    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:42.514320 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:42.513808    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:42.514320 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:42.514180    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:42.514320 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:42.514271    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:43.079235 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:43.079196    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:43.079686 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:43.079262    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:43.079686 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.079321    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:43.079686 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.079379    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:43.079686 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.079393    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret podName:0cf2fb43-34ef-4996-b746-31482ad247ab nodeName:}" failed. No retries permitted until 2026-09-10 20:58:51.07937233 +0000 UTC m=+18.081618656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret") pod "global-pull-secret-syncer-ks95v" (UID: "0cf2fb43-34ef-4996-b746-31482ad247ab") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:43.079686 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.079430    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs podName:9341a00d-f1e0-4798-b687-8f058c4e2d59 nodeName:}" failed. No retries permitted until 2026-09-10 20:58:51.079412174 +0000 UTC m=+18.081658514 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs") pod "network-metrics-daemon-8wqmx" (UID: "9341a00d-f1e0-4798-b687-8f058c4e2d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:43.180183 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:43.180149    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:43.180383 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.180253    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:43.180383 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.180279    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:43.180383 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.180293    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s9jqs for pod openshift-network-diagnostics/network-check-target-kd8gq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:43.180383 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.180348    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs podName:784ee7bb-d459-4694-b60e-10ef820b238a nodeName:}" failed. No retries permitted until 2026-09-10 20:58:51.180325981 +0000 UTC m=+18.182572310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9jqs" (UniqueName: "kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs") pod "network-check-target-kd8gq" (UID: "784ee7bb-d459-4694-b60e-10ef820b238a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:43.513933 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:43.513854    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:43.514156 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:43.513989    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:44.513773 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:44.513275    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:44.513773 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:44.513412    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:44.513773 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:44.513458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:44.513773 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:44.513568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:45.513003 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:45.512970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:45.513206 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:45.513092    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:46.513502 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:46.513470    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:46.513922 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:46.513585    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:46.513922 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:46.513663    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:46.513922 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:46.513788    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:47.513045 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:47.513017    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:47.513174 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:47.513148    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:48.512985 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:48.512948    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:48.513486 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:48.513069    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:48.513486 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:48.513146    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:48.513486 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:48.513256    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:49.512932 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:49.512900    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:49.513095 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:49.513033    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:50.512825 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:50.512793    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:50.513154 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:50.512908    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:50.513154 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:50.512962    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:50.513154 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:50.513050    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:51.143784 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:51.143749    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:51.143970 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:51.143804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:51.143970 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.143929    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:51.143970 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.143937    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:51.144110 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.143997    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs podName:9341a00d-f1e0-4798-b687-8f058c4e2d59 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:07.143979815 +0000 UTC m=+34.146226143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs") pod "network-metrics-daemon-8wqmx" (UID: "9341a00d-f1e0-4798-b687-8f058c4e2d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:58:51.144110 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.144011    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret podName:0cf2fb43-34ef-4996-b746-31482ad247ab nodeName:}" failed. No retries permitted until 2026-09-10 20:59:07.144005133 +0000 UTC m=+34.146251459 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret") pod "global-pull-secret-syncer-ks95v" (UID: "0cf2fb43-34ef-4996-b746-31482ad247ab") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:58:51.244975 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:51.244933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:51.245180 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.245114    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:58:51.245180 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.245154    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:58:51.245180 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.245167    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s9jqs for pod openshift-network-diagnostics/network-check-target-kd8gq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:51.245324 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.245225    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs podName:784ee7bb-d459-4694-b60e-10ef820b238a nodeName:}" failed. No retries permitted until 2026-09-10 20:59:07.245211283 +0000 UTC m=+34.247457622 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9jqs" (UniqueName: "kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs") pod "network-check-target-kd8gq" (UID: "784ee7bb-d459-4694-b60e-10ef820b238a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:58:51.513019 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:51.512943    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:51.513202 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:51.513082    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:52.512814 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:52.512772    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:52.512993 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:52.512913    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:52.513283 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:52.513263    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:52.513614 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:52.513342    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:53.514067 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:53.514033    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:53.514549 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:53.514150    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:54.513454 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:54.513416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:54.513454 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:54.513453    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:54.513661 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:54.513568    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:54.513722 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:54.513672    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:55.516134 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.515876    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:55.516763 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:55.516264    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:55.726143 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.726051    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"52a507a7ba0d7fcb059a4b4424dbaa0a1db274ec5a107d0e0083955b8e116d19"}
Sep 10 20:58:55.726385 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.726155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"9f587e0cbb4f1e953932d7cc7a2bddbb4cf7c2b546ad2597d5470dfb859e4957"}
Sep 10 20:58:55.726385 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.726209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"e50ae02a31d9c7d3ba2f53d06d958366b17e616b52d5d86992e5042ab545e217"}
Sep 10 20:58:55.726385 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.726250    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"01eae22fd9cd2d90653a411ce16708f6ded7fa0077e26ec62d0754a6e9e40ebb"}
Sep 10 20:58:55.726385 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.726296    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"aeaf80825feedb2e28a3bcc925ef67a078e73518c860d77bc11f81b05cd60e05"}
Sep 10 20:58:55.726385 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.726342    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"ad1545c1946573beaac80856c6b58cf1e21645e49159b22f1bc833ed5c59a156"}
Sep 10 20:58:55.727536 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.727510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj" event={"ID":"10564b7c-e01c-4287-9ecf-f01ebfbfe70c","Type":"ContainerStarted","Data":"37a6b706d0f4ccc92a2a95ddecbe2738133f1a25708d615299cb0f1fa159ab59"}
Sep 10 20:58:55.728897 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.728865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lfjp5" event={"ID":"857fc54f-2c08-4613-84d4-40a4c8b3152a","Type":"ContainerStarted","Data":"ca9ecbbcff1e4ba353d0ddfed0f352a832dbd39dc8b6cb381a0bfbaf0849109a"}
Sep 10 20:58:55.730261 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.730232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xcgkb" event={"ID":"4a77b4af-edcb-478c-8d57-4b7b29339a22","Type":"ContainerStarted","Data":"009925a042a1776a9bf4f4b45138f0de4f1754331b11db9c5e807a180216bf4c"}
Sep 10 20:58:55.731751 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.731727    2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25" containerID="6d4feec1df11b2a7c63f9f2edd05840354b92b857e94e5707cf3f44bd7ec8457" exitCode=0
Sep 10 20:58:55.731830 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.731807    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerDied","Data":"6d4feec1df11b2a7c63f9f2edd05840354b92b857e94e5707cf3f44bd7ec8457"}
Sep 10 20:58:55.733337 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.733316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gp8hn" event={"ID":"7ff24c61-0ba5-4a58-a41c-95284c27b8d3","Type":"ContainerStarted","Data":"1dc9fa79440891ed87962f0cc45f1ce15d59833fb80133db7aa02339b7c62a1c"}
Sep 10 20:58:55.734838 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.734815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wq6lz" event={"ID":"0b7e62b8-d8b2-431e-8d20-27577cd0f03c","Type":"ContainerStarted","Data":"4dab86f55a280c5c54273b44efd24db67def030bb8b0054b98e6cc78c122dd50"}
Sep 10 20:58:55.736446 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.736421    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctnsl" event={"ID":"372eb974-c525-4f33-954a-23c699ac2016","Type":"ContainerStarted","Data":"dc30b619356af6163634598175fbca8aac9a1f5daf2a7e1019b5ff14b40df28b"}
Sep 10 20:58:55.748783 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.746772    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-236.ec2.internal" podStartSLOduration=21.746754213 podStartE2EDuration="21.746754213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:38.70199425 +0000 UTC m=+5.704240591" watchObservedRunningTime="2026-09-10 20:58:55.746754213 +0000 UTC m=+22.749000561"
Sep 10 20:58:55.768849 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.768803    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lfjp5" podStartSLOduration=4.495241528 podStartE2EDuration="22.768785678s" totalImagesPullingTime="18.27354415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:55.74918177 +0000 UTC m=+22.751428116" watchObservedRunningTime="2026-09-10 20:58:55.768785678 +0000 UTC m=+22.771032027"
Sep 10 20:58:55.769250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.769224    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wq6lz" podStartSLOduration=11.823578732 podStartE2EDuration="22.769210247s" totalImagesPullingTime="10.945631515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:55.768606956 +0000 UTC m=+22.770853306" watchObservedRunningTime="2026-09-10 20:58:55.769210247 +0000 UTC m=+22.771456599"
Sep 10 20:58:55.805660 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.805611    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gp8hn" podStartSLOduration=4.548561368 podStartE2EDuration="22.805596114s" totalImagesPullingTime="18.257034746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:55.805373386 +0000 UTC m=+22.807619735" watchObservedRunningTime="2026-09-10 20:58:55.805596114 +0000 UTC m=+22.807842461"
Sep 10 20:58:55.824317 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.824274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xcgkb" podStartSLOduration=4.490200809 podStartE2EDuration="22.824257948s" totalImagesPullingTime="18.334057139s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:55.823897571 +0000 UTC m=+22.826143924" watchObservedRunningTime="2026-09-10 20:58:55.824257948 +0000 UTC m=+22.826504295"
Sep 10 20:58:55.838201 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.838155    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ctnsl" podStartSLOduration=4.576771853 podStartE2EDuration="22.838135767s" totalImagesPullingTime="18.261363914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:55.837719999 +0000 UTC m=+22.839966340" watchObservedRunningTime="2026-09-10 20:58:55.838135767 +0000 UTC m=+22.840382108"
Sep 10 20:58:55.905138 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.905098    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:55.905661 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:55.905644    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:56.513884 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:56.513625    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:56.513993 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:56.513924    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:56.513993 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:56.513624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:56.514059 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:56.514004    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:56.569226 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:56.569200    2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 20:58:56.740346 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:56.740254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lnz7h" event={"ID":"717d024e-fb07-41d1-b9d2-f2180c6bad68","Type":"ContainerStarted","Data":"6aea6e9350c03c4a81abbf9c68e50584286323e30023e9ee06a4e16c3305fbdb"}
Sep 10 20:58:56.742637 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:56.742570    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj" event={"ID":"10564b7c-e01c-4287-9ecf-f01ebfbfe70c","Type":"ContainerStarted","Data":"650c128c4ddcd33110ed86c8dee624b8be191bdd70317be402743964607d44cc"}
Sep 10 20:58:56.743142 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:56.743099    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:56.743598 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:56.743578    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wq6lz"
Sep 10 20:58:56.772313 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:56.772274    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lnz7h" podStartSLOduration=5.515962575 podStartE2EDuration="23.772260467s" totalImagesPullingTime="18.256297892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:56.756029609 +0000 UTC m=+23.758275956" watchObservedRunningTime="2026-09-10 20:58:56.772260467 +0000 UTC m=+23.774506813"
Sep 10 20:58:57.473699 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:57.473584    2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T20:58:56.569216765Z","UUID":"e30fb23a-8001-4d1b-8e13-ffab0bb28d5a","Handler":null,"Name":"","Endpoint":""}
Sep 10 20:58:57.475891 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:57.475868    2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 20:58:57.476020 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:57.475904    2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 20:58:57.515444 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:57.515422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:57.515552 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:57.515521    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:58:57.750816 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:57.750736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"3403fa0e630bbd777170f2d3bc597c25fa9e84acca341cd41e6cd61742fc8f12"}
Sep 10 20:58:57.752996 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:57.752963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj" event={"ID":"10564b7c-e01c-4287-9ecf-f01ebfbfe70c","Type":"ContainerStarted","Data":"af4e8187076e047c3541419fc44705d55e3b04da7cdee39b377935ea463ba0f7"}
Sep 10 20:58:57.774304 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:57.774248    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fwqhj" podStartSLOduration=4.142423205 podStartE2EDuration="24.77423445s" totalImagesPullingTime="20.631811245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:58:57.774182621 +0000 UTC m=+24.776428972" watchObservedRunningTime="2026-09-10 20:58:57.77423445 +0000 UTC m=+24.776480837"
Sep 10 20:58:58.513004 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:58.512968    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:58:58.513204 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:58.513080    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:58:58.513204 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:58.513139    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:58:58.513336 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:58.513251    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:58:59.512822 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:58:59.512790    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:58:59.513445 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:58:59.512924    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:59:00.513132 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:00.513082    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:00.513621 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:00.513206    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:59:00.513621 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:00.513257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:00.513621 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:00.513368    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:59:00.763792 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:00.763546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" event={"ID":"ad2e3059-c7d0-405d-8ff2-8a97f0644f81","Type":"ContainerStarted","Data":"7f596cb161bd654fe4a5d9dba21b183e0a359598940874db016fe84d40ca6aee"}
Sep 10 20:59:00.763879 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:00.763811    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:59:00.763879 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:00.763868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:59:00.779490 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:00.779466    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:59:00.779620 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:00.779575    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:59:00.791634 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:00.791592    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" podStartSLOduration=9.460312602 podStartE2EDuration="27.791578723s" totalImagesPullingTime="18.331266121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:00.790327521 +0000 UTC m=+27.792573868" watchObservedRunningTime="2026-09-10 20:59:00.791578723 +0000 UTC m=+27.793825069"
Sep 10 20:59:01.516494 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:01.516466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:59:01.516925 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:01.516561    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:59:01.768148 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:01.768046    2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25" containerID="c17e39a2db9c35c8116a5894f189824bd55f8d4c73b3737bb369f8293f1e9f77" exitCode=0
Sep 10 20:59:01.768397 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:01.768152    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerDied","Data":"c17e39a2db9c35c8116a5894f189824bd55f8d4c73b3737bb369f8293f1e9f77"}
Sep 10 20:59:01.768397 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:01.768268    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 20:59:02.513219 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.513028    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:02.513330 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.513044    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:02.513395 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:02.513323    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:59:02.513395 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:02.513383    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:59:02.596238 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.596209    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kd8gq"]
Sep 10 20:59:02.597107 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.596984    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ks95v"]
Sep 10 20:59:02.597786 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.597763    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8wqmx"]
Sep 10 20:59:02.597910 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.597881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:59:02.598014 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:02.597993    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:59:02.772742 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.772714    2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25" containerID="55afc08333b0077ef65e19856b0d0b73301b84b18ac53760aa605c88c0082d44" exitCode=0
Sep 10 20:59:02.772888 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.772822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerDied","Data":"55afc08333b0077ef65e19856b0d0b73301b84b18ac53760aa605c88c0082d44"}
Sep 10 20:59:02.772888 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.772835    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:02.773018 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.772975    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 20:59:02.773018 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:02.772983    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:02.773380 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:02.773354    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:59:02.773490 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:02.773436    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:59:03.777513 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:03.777485    2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25" containerID="f243c791cc896b966a504748ca8b696e84eb166631130aca7b68bf1af0520073" exitCode=0
Sep 10 20:59:03.777925 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:03.777554    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerDied","Data":"f243c791cc896b966a504748ca8b696e84eb166631130aca7b68bf1af0520073"}
Sep 10 20:59:04.513752 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:04.513678    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:04.513887 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:04.513680    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:04.513887 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:04.513777    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:59:04.513887 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:04.513805    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:59:04.513998 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:04.513882    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:59:04.513998 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:04.513936    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:59:05.231584 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:05.231548    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:59:05.232111 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:05.231786    2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 10 20:59:05.243020 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:05.242986    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" podUID="ad2e3059-c7d0-405d-8ff2-8a97f0644f81" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 20:59:05.253534 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:05.253216    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx" podUID="ad2e3059-c7d0-405d-8ff2-8a97f0644f81" containerName="ovnkube-controller" probeResult="failure" output=""
Sep 10 20:59:06.513616 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:06.513584    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:06.514009 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:06.513587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:59:06.514009 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:06.513702    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:59:06.514009 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:06.513587    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:06.514009 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:06.513768    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:59:06.514009 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:06.513864    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:59:07.157764 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:07.157730    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:59:07.157953 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:07.157812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:07.157953 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:07.157902    2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:07.157953 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:07.157902    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:07.158141 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:07.157969    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret podName:0cf2fb43-34ef-4996-b746-31482ad247ab nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.157950608 +0000 UTC m=+66.160196953 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret") pod "global-pull-secret-syncer-ks95v" (UID: "0cf2fb43-34ef-4996-b746-31482ad247ab") : object "kube-system"/"original-pull-secret" not registered
Sep 10 20:59:07.158141 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:07.157988    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs podName:9341a00d-f1e0-4798-b687-8f058c4e2d59 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.157979387 +0000 UTC m=+66.160225726 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs") pod "network-metrics-daemon-8wqmx" (UID: "9341a00d-f1e0-4798-b687-8f058c4e2d59") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 20:59:07.258523 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:07.258476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:07.258682 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:07.258629    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 20:59:07.258682 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:07.258649    2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 20:59:07.258682 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:07.258658    2580 projected.go:196] Error preparing data for projected volume kube-api-access-s9jqs for pod openshift-network-diagnostics/network-check-target-kd8gq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:07.258850 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:07.258703    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs podName:784ee7bb-d459-4694-b60e-10ef820b238a nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.258690753 +0000 UTC m=+66.260937090 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-s9jqs" (UniqueName: "kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs") pod "network-check-target-kd8gq" (UID: "784ee7bb-d459-4694-b60e-10ef820b238a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 20:59:08.513738 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.513562    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:08.514187 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.513567    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:59:08.514187 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:08.513816    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ks95v" podUID="0cf2fb43-34ef-4996-b746-31482ad247ab"
Sep 10 20:59:08.514187 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:08.513891    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8wqmx" podUID="9341a00d-f1e0-4798-b687-8f058c4e2d59"
Sep 10 20:59:08.514187 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.513571    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:08.514187 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:08.513971    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kd8gq" podUID="784ee7bb-d459-4694-b60e-10ef820b238a"
Sep 10 20:59:08.786952 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.786929    2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-236.ec2.internal" event="NodeReady"
Sep 10 20:59:08.787062 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.787050    2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 20:59:08.791762 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.791736    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerStarted","Data":"c291d49b0dafc6b967d33bc9dc004833c24ee946dfc3bb11b91ce15a2a01540f"}
Sep 10 20:59:08.858891 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.858861    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-l68d2"]
Sep 10 20:59:08.877247 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.877223    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fwkbp"]
Sep 10 20:59:08.877360 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.877342    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:08.882414 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.882330    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 20:59:08.882414 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.882343    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 20:59:08.882414 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.882332    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7j9n7\""
Sep 10 20:59:08.896831 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.896809    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l68d2"]
Sep 10 20:59:08.896928 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.896849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fwkbp"]
Sep 10 20:59:08.896928 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.896912    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fwkbp"
Sep 10 20:59:08.906052 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.906030    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9vpqg\""
Sep 10 20:59:08.906161 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.906133    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 20:59:08.906410 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.906395    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 20:59:08.906477 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.906414    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 20:59:08.970538 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.970508    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtx7l\" (UniqueName: \"kubernetes.io/projected/2ba3f4ef-5e26-47d9-92b0-37bae0dd5650-kube-api-access-vtx7l\") pod \"ingress-canary-fwkbp\" (UID: \"2ba3f4ef-5e26-47d9-92b0-37bae0dd5650\") " pod="openshift-ingress-canary/ingress-canary-fwkbp"
Sep 10 20:59:08.970649 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.970555    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/107d30b3-f5f5-455b-8834-509a7b83772f-config-volume\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:08.970649 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.970577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krl65\" (UniqueName: \"kubernetes.io/projected/107d30b3-f5f5-455b-8834-509a7b83772f-kube-api-access-krl65\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:08.970746 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.970663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:08.970746 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.970687    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ba3f4ef-5e26-47d9-92b0-37bae0dd5650-cert\") pod \"ingress-canary-fwkbp\" (UID: \"2ba3f4ef-5e26-47d9-92b0-37bae0dd5650\") " pod="openshift-ingress-canary/ingress-canary-fwkbp"
Sep 10 20:59:08.970746 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:08.970703    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/107d30b3-f5f5-455b-8834-509a7b83772f-tmp-dir\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.071573 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.071546    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ba3f4ef-5e26-47d9-92b0-37bae0dd5650-cert\") pod \"ingress-canary-fwkbp\" (UID: \"2ba3f4ef-5e26-47d9-92b0-37bae0dd5650\") " pod="openshift-ingress-canary/ingress-canary-fwkbp"
Sep 10 20:59:09.071573 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.071578    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/107d30b3-f5f5-455b-8834-509a7b83772f-tmp-dir\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.071799 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.071611    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtx7l\" (UniqueName: \"kubernetes.io/projected/2ba3f4ef-5e26-47d9-92b0-37bae0dd5650-kube-api-access-vtx7l\") pod \"ingress-canary-fwkbp\" (UID: \"2ba3f4ef-5e26-47d9-92b0-37bae0dd5650\") " pod="openshift-ingress-canary/ingress-canary-fwkbp"
Sep 10 20:59:09.071799 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.071651    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/107d30b3-f5f5-455b-8834-509a7b83772f-config-volume\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.071799 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.071771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krl65\" (UniqueName: \"kubernetes.io/projected/107d30b3-f5f5-455b-8834-509a7b83772f-kube-api-access-krl65\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.071943 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.071860    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.072001 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:09.071954    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:09.072051 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:09.072000    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls podName:107d30b3-f5f5-455b-8834-509a7b83772f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:09.571986096 +0000 UTC m=+36.574232420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls") pod "dns-default-l68d2" (UID: "107d30b3-f5f5-455b-8834-509a7b83772f") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:09.072051 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.072009    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/107d30b3-f5f5-455b-8834-509a7b83772f-tmp-dir\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.072256 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.072236    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/107d30b3-f5f5-455b-8834-509a7b83772f-config-volume\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.076030 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.076005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2ba3f4ef-5e26-47d9-92b0-37bae0dd5650-cert\") pod \"ingress-canary-fwkbp\" (UID: \"2ba3f4ef-5e26-47d9-92b0-37bae0dd5650\") " pod="openshift-ingress-canary/ingress-canary-fwkbp"
Sep 10 20:59:09.087250 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.087233    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtx7l\" (UniqueName: \"kubernetes.io/projected/2ba3f4ef-5e26-47d9-92b0-37bae0dd5650-kube-api-access-vtx7l\") pod \"ingress-canary-fwkbp\" (UID: \"2ba3f4ef-5e26-47d9-92b0-37bae0dd5650\") " pod="openshift-ingress-canary/ingress-canary-fwkbp"
Sep 10 20:59:09.090999 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.090982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krl65\" (UniqueName: \"kubernetes.io/projected/107d30b3-f5f5-455b-8834-509a7b83772f-kube-api-access-krl65\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.232467 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.232400    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fwkbp"
Sep 10 20:59:09.567371 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.567301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fwkbp"]
Sep 10 20:59:09.573905 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:59:09.573881    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba3f4ef_5e26_47d9_92b0_37bae0dd5650.slice/crio-d8c88d7df7c18bb08657ed35ed236e078a654155ea7db0058ae803b912b86baa WatchSource:0}: Error finding container d8c88d7df7c18bb08657ed35ed236e078a654155ea7db0058ae803b912b86baa: Status 404 returned error can't find the container with id d8c88d7df7c18bb08657ed35ed236e078a654155ea7db0058ae803b912b86baa
Sep 10 20:59:09.574870 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.574852    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:09.574958 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:09.574946    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:09.575018 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:09.575006    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls podName:107d30b3-f5f5-455b-8834-509a7b83772f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:10.574989655 +0000 UTC m=+37.577235995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls") pod "dns-default-l68d2" (UID: "107d30b3-f5f5-455b-8834-509a7b83772f") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:09.795427 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.795388    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fwkbp" event={"ID":"2ba3f4ef-5e26-47d9-92b0-37bae0dd5650","Type":"ContainerStarted","Data":"d8c88d7df7c18bb08657ed35ed236e078a654155ea7db0058ae803b912b86baa"}
Sep 10 20:59:09.798184 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.798158    2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25" containerID="c291d49b0dafc6b967d33bc9dc004833c24ee946dfc3bb11b91ce15a2a01540f" exitCode=0
Sep 10 20:59:09.798317 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:09.798207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerDied","Data":"c291d49b0dafc6b967d33bc9dc004833c24ee946dfc3bb11b91ce15a2a01540f"}
Sep 10 20:59:10.512917 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.512884    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:10.512917 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.512918    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:10.513173 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.512888    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:59:10.515651 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.515576    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:59:10.515651 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.515581    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:59:10.516574 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.516556    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wb7zl\""
Sep 10 20:59:10.516691 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.516583    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:59:10.516760 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.516710    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5khf6\""
Sep 10 20:59:10.516815 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.516785    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:59:10.582470 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.582440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:10.582791 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:10.582551    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:10.582791 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:10.582611    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls podName:107d30b3-f5f5-455b-8834-509a7b83772f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:12.582591654 +0000 UTC m=+39.584837979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls") pod "dns-default-l68d2" (UID: "107d30b3-f5f5-455b-8834-509a7b83772f") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:10.805529 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.805264    2580 generic.go:356] "Generic (PLEG): container finished" podID="1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25" containerID="3d638fd30601fc29745314a815d0a6fd6cb05714da125758bdb7e1a446c242c0" exitCode=0
Sep 10 20:59:10.805529 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:10.805316    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerDied","Data":"3d638fd30601fc29745314a815d0a6fd6cb05714da125758bdb7e1a446c242c0"}
Sep 10 20:59:11.811642 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:11.811605    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" event={"ID":"1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25","Type":"ContainerStarted","Data":"3c4a6e6bf732b5985f6a415b3bc9368ad8a8f8717b55d066df438a5b8bdca672"}
Sep 10 20:59:11.837948 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:11.837897    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pcz7g" podStartSLOduration=9.348299097 podStartE2EDuration="38.837882554s" totalImagesPullingTime="29.489583457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:11.836585314 +0000 UTC m=+38.838831660" watchObservedRunningTime="2026-09-10 20:59:11.837882554 +0000 UTC m=+38.840128897"
Sep 10 20:59:12.595623 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:12.595592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:12.595768 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:12.595738    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:12.595812 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:12.595801    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls podName:107d30b3-f5f5-455b-8834-509a7b83772f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:16.595784368 +0000 UTC m=+43.598030695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls") pod "dns-default-l68d2" (UID: "107d30b3-f5f5-455b-8834-509a7b83772f") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:12.815875 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:12.815841    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fwkbp" event={"ID":"2ba3f4ef-5e26-47d9-92b0-37bae0dd5650","Type":"ContainerStarted","Data":"9efc028f37fe3cd29e69472fef7151e3ae1b01eba2825e68fb0a30e9b20bd363"}
Sep 10 20:59:16.622427 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:16.622381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:16.622944 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:16.622512    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:16.622944 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:16.622567    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls podName:107d30b3-f5f5-455b-8834-509a7b83772f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:24.622552619 +0000 UTC m=+51.624798944 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls") pod "dns-default-l68d2" (UID: "107d30b3-f5f5-455b-8834-509a7b83772f") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:24.680645 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:24.680600    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:24.681044 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:24.680725    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:24.681044 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:24.680781    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls podName:107d30b3-f5f5-455b-8834-509a7b83772f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:40.680765588 +0000 UTC m=+67.683011936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls") pod "dns-default-l68d2" (UID: "107d30b3-f5f5-455b-8834-509a7b83772f") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:35.253230 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:35.253199    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2dnpx"
Sep 10 20:59:35.292500 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:35.292447    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fwkbp" podStartSLOduration=24.867683493 podStartE2EDuration="27.292427995s" totalImagesPullingTime="2.424744502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:12.845348652 +0000 UTC m=+39.847594999" watchObservedRunningTime="2026-09-10 20:59:35.292427995 +0000 UTC m=+62.294674341"
Sep 10 20:59:38.849040 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.849004    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"]
Sep 10 20:59:38.879817 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.879786    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r"]
Sep 10 20:59:38.879967 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.879895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:38.882288 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.882266    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 20:59:38.882411 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.882338    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d6dsj\""
Sep 10 20:59:38.882457 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.882266    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 20:59:38.882550 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.882532    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:59:38.882618 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.882569    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 20:59:38.901772 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.901756    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r"]
Sep 10 20:59:38.901872 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.901780    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"]
Sep 10 20:59:38.901872 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.901846    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r"
Sep 10 20:59:38.905062 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.905045    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-4jr52\""
Sep 10 20:59:38.905171 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.905082    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Sep 10 20:59:38.905171 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.905092    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Sep 10 20:59:38.955852 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.954465    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7bb47ccbbb-cpmj2"]
Sep 10 20:59:38.971974 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.971951    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:38.973299 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.973279    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7bb47ccbbb-cpmj2"]
Sep 10 20:59:38.974150 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.974114    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-qd5p5\""
Sep 10 20:59:38.974228 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.974199    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 20:59:38.974731 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.974714    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 10 20:59:38.975159 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.975143    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 10 20:59:38.975251 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.975150    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 20:59:38.975840 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.975824    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 10 20:59:38.975938 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.975922    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 10 20:59:38.978439 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.978416    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26a53a6f-5cc0-4659-ad6d-cc250ea634af-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:38.978549 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.978452    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2h5t\" (UniqueName: \"kubernetes.io/projected/26a53a6f-5cc0-4659-ad6d-cc250ea634af-kube-api-access-h2h5t\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:38.978549 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:38.978528    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26a53a6f-5cc0-4659-ad6d-cc250ea634af-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:39.047497 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.047476    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-78d9ff86dd-646b7"]
Sep 10 20:59:39.067928 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.067908    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-78d9ff86dd-646b7"]
Sep 10 20:59:39.068028 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.067994    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.070500 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.070476    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 20:59:39.070768 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.070752    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-k4c27\""
Sep 10 20:59:39.070872 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.070787    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 20:59:39.070976 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.070962    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 20:59:39.077200 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.077183    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 20:59:39.079556 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079540    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnpqf\" (UniqueName: \"kubernetes.io/projected/ee33f0ef-90c8-4e2c-b34d-0459c159036d-kube-api-access-fnpqf\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.079611 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-bound-sa-token\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.079611 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079588    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-stats-auth\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.079685 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26a53a6f-5cc0-4659-ad6d-cc250ea634af-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:39.079685 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079653    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2h5t\" (UniqueName: \"kubernetes.io/projected/26a53a6f-5cc0-4659-ad6d-cc250ea634af-kube-api-access-h2h5t\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:39.079747 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079688    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldr6m\" (UniqueName: \"kubernetes.io/projected/72eb50b8-401f-4e0a-8416-1901732a1e19-kube-api-access-ldr6m\") pod \"volume-data-source-validator-86695b7cc7-l726r\" (UID: \"72eb50b8-401f-4e0a-8416-1901732a1e19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r"
Sep 10 20:59:39.079747 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079706    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.079747 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079723    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.079889 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079750    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-image-registry-private-configuration\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.079889 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079813    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-certificates\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.079889 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079848    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-installation-pull-secrets\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.079994 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88t2\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-kube-api-access-c88t2\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.079994 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079913    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-trusted-ca\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.079994 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.079994 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.079973    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-default-certificate\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.080166 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.080035    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ac41557-4731-47b3-93d2-f98b5b67e55f-ca-trust-extracted\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.080166 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.080061    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26a53a6f-5cc0-4659-ad6d-cc250ea634af-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:39.080478 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.080464    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/26a53a6f-5cc0-4659-ad6d-cc250ea634af-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:39.082890 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.082875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/26a53a6f-5cc0-4659-ad6d-cc250ea634af-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:39.099746 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.099698    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2h5t\" (UniqueName: \"kubernetes.io/projected/26a53a6f-5cc0-4659-ad6d-cc250ea634af-kube-api-access-h2h5t\") pod \"kube-storage-version-migrator-operator-6c8f84f897-4f6l7\" (UID: \"26a53a6f-5cc0-4659-ad6d-cc250ea634af\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:39.180800 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.180775    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.180888 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.180807    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-default-certificate\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.180888 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.180827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ac41557-4731-47b3-93d2-f98b5b67e55f-ca-trust-extracted\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.180957 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.180925    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:59:39.180957 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.180940    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78d9ff86dd-646b7: secret "image-registry-tls" not found
Sep 10 20:59:39.181021 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.180992    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls podName:7ac41557-4731-47b3-93d2-f98b5b67e55f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.680972591 +0000 UTC m=+66.683218938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls") pod "image-registry-78d9ff86dd-646b7" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f") : secret "image-registry-tls" not found
Sep 10 20:59:39.181062 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181033    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnpqf\" (UniqueName: \"kubernetes.io/projected/ee33f0ef-90c8-4e2c-b34d-0459c159036d-kube-api-access-fnpqf\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.181223 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-bound-sa-token\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.181223 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:39.181223 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181153    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-stats-auth\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.181223 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldr6m\" (UniqueName: \"kubernetes.io/projected/72eb50b8-401f-4e0a-8416-1901732a1e19-kube-api-access-ldr6m\") pod \"volume-data-source-validator-86695b7cc7-l726r\" (UID: \"72eb50b8-401f-4e0a-8416-1901732a1e19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r"
Sep 10 20:59:39.181477 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.181477 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181264    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.181477 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181295    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-image-registry-private-configuration\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.181477 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181321    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-certificates\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.181477 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.181391    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:59:39.181477 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.181442    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.681425973 +0000 UTC m=+66.683672315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : secret "router-metrics-certs-default" not found
Sep 10 20:59:39.181477 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.181463    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:39.681450263 +0000 UTC m=+66.683696606 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : configmap references non-existent config key: service-ca.crt
Sep 10 20:59:39.181839 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181517    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-installation-pull-secrets\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.181839 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181541    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88t2\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-kube-api-access-c88t2\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.181839 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181568    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 20:59:39.181839 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.181758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-trusted-ca\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.182214 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.182007    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ac41557-4731-47b3-93d2-f98b5b67e55f-ca-trust-extracted\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.182214 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.182053    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-certificates\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.183274 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.183243    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-trusted-ca\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.183709 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.183326    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 20:59:39.183709 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.183334    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 20:59:39.183709 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.183631    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-default-certificate\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.183835 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.183716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-stats-auth\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.183835 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.183799    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-image-registry-private-configuration\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.184099 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.184080    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-installation-pull-secrets\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.188244 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.188228    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"
Sep 10 20:59:39.191014 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.190991    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88t2\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-kube-api-access-c88t2\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.191191 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.191173    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-bound-sa-token\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.191334 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.191317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldr6m\" (UniqueName: \"kubernetes.io/projected/72eb50b8-401f-4e0a-8416-1901732a1e19-kube-api-access-ldr6m\") pod \"volume-data-source-validator-86695b7cc7-l726r\" (UID: \"72eb50b8-401f-4e0a-8416-1901732a1e19\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r"
Sep 10 20:59:39.191513 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.191495    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnpqf\" (UniqueName: \"kubernetes.io/projected/ee33f0ef-90c8-4e2c-b34d-0459c159036d-kube-api-access-fnpqf\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.191676 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.191663    2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 20:59:39.191735 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.191713    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs podName:9341a00d-f1e0-4798-b687-8f058c4e2d59 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:43.191697072 +0000 UTC m=+130.193943398 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs") pod "network-metrics-daemon-8wqmx" (UID: "9341a00d-f1e0-4798-b687-8f058c4e2d59") : secret "metrics-daemon-secret" not found
Sep 10 20:59:39.193238 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.193218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0cf2fb43-34ef-4996-b746-31482ad247ab-original-pull-secret\") pod \"global-pull-secret-syncer-ks95v\" (UID: \"0cf2fb43-34ef-4996-b746-31482ad247ab\") " pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:39.210181 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.210165    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r"
Sep 10 20:59:39.283462 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.283164    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:39.285969 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.285946    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 20:59:39.296133 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.296101    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 20:59:39.307299 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.307280    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jqs\" (UniqueName: \"kubernetes.io/projected/784ee7bb-d459-4694-b60e-10ef820b238a-kube-api-access-s9jqs\") pod \"network-check-target-kd8gq\" (UID: \"784ee7bb-d459-4694-b60e-10ef820b238a\") " pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:39.324174 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.324154    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ks95v"
Sep 10 20:59:39.338948 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.338935    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wb7zl\""
Sep 10 20:59:39.346967 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.346949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:39.478947 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.478911    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7"]
Sep 10 20:59:39.483478 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:59:39.483447    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a53a6f_5cc0_4659_ad6d_cc250ea634af.slice/crio-4d236a5e3bf172f88e93d8078920d4f5a04a332e8e615f5afa17ddc48890cbd1 WatchSource:0}: Error finding container 4d236a5e3bf172f88e93d8078920d4f5a04a332e8e615f5afa17ddc48890cbd1: Status 404 returned error can't find the container with id 4d236a5e3bf172f88e93d8078920d4f5a04a332e8e615f5afa17ddc48890cbd1
Sep 10 20:59:39.503326 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.503303    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r"]
Sep 10 20:59:39.505732 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:59:39.505711    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72eb50b8_401f_4e0a_8416_1901732a1e19.slice/crio-7b71bd66e340e5e44cf3d979ee2ca997a49399ee2907d5c19a2d53edb8715c58 WatchSource:0}: Error finding container 7b71bd66e340e5e44cf3d979ee2ca997a49399ee2907d5c19a2d53edb8715c58: Status 404 returned error can't find the container with id 7b71bd66e340e5e44cf3d979ee2ca997a49399ee2907d5c19a2d53edb8715c58
Sep 10 20:59:39.611237 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.611173    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ks95v"]
Sep 10 20:59:39.614160 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:59:39.614113    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf2fb43_34ef_4996_b746_31482ad247ab.slice/crio-cd0169a2c199cd286ae80d4e749cc2b4f5cca91c44f8759a76708447a5ddf9a9 WatchSource:0}: Error finding container cd0169a2c199cd286ae80d4e749cc2b4f5cca91c44f8759a76708447a5ddf9a9: Status 404 returned error can't find the container with id cd0169a2c199cd286ae80d4e749cc2b4f5cca91c44f8759a76708447a5ddf9a9
Sep 10 20:59:39.641109 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.641084    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kd8gq"]
Sep 10 20:59:39.645028 ip-10-0-128-236 kubenswrapper[2580]: W0910 20:59:39.645005    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784ee7bb_d459_4694_b60e_10ef820b238a.slice/crio-95231a923202c777dba1f24dd5fa67363d032b46b4a261633198aaa184b3252b WatchSource:0}: Error finding container 95231a923202c777dba1f24dd5fa67363d032b46b4a261633198aaa184b3252b: Status 404 returned error can't find the container with id 95231a923202c777dba1f24dd5fa67363d032b46b4a261633198aaa184b3252b
Sep 10 20:59:39.685225 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.685205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.685321 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.685232    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:39.685321 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.685261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:39.685390 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.685359    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:40.685341995 +0000 UTC m=+67.687588320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : configmap references non-existent config key: service-ca.crt
Sep 10 20:59:39.685390 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.685363    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:59:39.685453 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.685367    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:59:39.685453 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.685432    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78d9ff86dd-646b7: secret "image-registry-tls" not found
Sep 10 20:59:39.685514 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.685418    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:40.685400513 +0000 UTC m=+67.687646842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : secret "router-metrics-certs-default" not found
Sep 10 20:59:39.685514 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:39.685470    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls podName:7ac41557-4731-47b3-93d2-f98b5b67e55f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:40.685461064 +0000 UTC m=+67.687707389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls") pod "image-registry-78d9ff86dd-646b7" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f") : secret "image-registry-tls" not found
Sep 10 20:59:39.876999 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.876967    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kd8gq" event={"ID":"784ee7bb-d459-4694-b60e-10ef820b238a","Type":"ContainerStarted","Data":"95231a923202c777dba1f24dd5fa67363d032b46b4a261633198aaa184b3252b"}
Sep 10 20:59:39.878167 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.878148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ks95v" event={"ID":"0cf2fb43-34ef-4996-b746-31482ad247ab","Type":"ContainerStarted","Data":"cd0169a2c199cd286ae80d4e749cc2b4f5cca91c44f8759a76708447a5ddf9a9"}
Sep 10 20:59:39.879282 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.879264    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r" event={"ID":"72eb50b8-401f-4e0a-8416-1901732a1e19","Type":"ContainerStarted","Data":"7b71bd66e340e5e44cf3d979ee2ca997a49399ee2907d5c19a2d53edb8715c58"}
Sep 10 20:59:39.880371 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:39.880352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7" event={"ID":"26a53a6f-5cc0-4659-ad6d-cc250ea634af","Type":"ContainerStarted","Data":"4d236a5e3bf172f88e93d8078920d4f5a04a332e8e615f5afa17ddc48890cbd1"}
Sep 10 20:59:40.695731 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:40.695690    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 20:59:40.696052 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:40.695755    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:40.696052 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:40.695789    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:40.696052 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:40.695840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:40.696052 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:40.695875    2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 20:59:40.696052 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:40.695957    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls podName:107d30b3-f5f5-455b-8834-509a7b83772f nodeName:}" failed. No retries permitted until 2026-09-10 21:00:12.695933956 +0000 UTC m=+99.698180300 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls") pod "dns-default-l68d2" (UID: "107d30b3-f5f5-455b-8834-509a7b83772f") : secret "dns-default-metrics-tls" not found
Sep 10 20:59:40.696052 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:40.695996    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:59:40.696052 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:40.696010    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78d9ff86dd-646b7: secret "image-registry-tls" not found
Sep 10 20:59:40.696489 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:40.696071    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls podName:7ac41557-4731-47b3-93d2-f98b5b67e55f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.696055964 +0000 UTC m=+69.698302292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls") pod "image-registry-78d9ff86dd-646b7" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f") : secret "image-registry-tls" not found
Sep 10 20:59:40.696489 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:40.696167    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.696155658 +0000 UTC m=+69.698401986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : configmap references non-existent config key: service-ca.crt
Sep 10 20:59:40.696489 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:40.696234    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:59:40.696489 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:40.696266    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:42.696254787 +0000 UTC m=+69.698501115 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : secret "router-metrics-certs-default" not found
Sep 10 20:59:41.889180 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:41.888755    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r" event={"ID":"72eb50b8-401f-4e0a-8416-1901732a1e19","Type":"ContainerStarted","Data":"2cd9f8ef724a165e8beab3fb2a228c8241c52b541cec1986a7eafd3e9f84ac36"}
Sep 10 20:59:41.904706 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:41.903911    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-l726r" podStartSLOduration=2.587804349 podStartE2EDuration="3.903891457s" totalImagesPullingTime="1.316087108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:41.903396484 +0000 UTC m=+68.905642831" watchObservedRunningTime="2026-09-10 20:59:41.903891457 +0000 UTC m=+68.906137802"
Sep 10 20:59:42.712452 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:42.712408    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:42.712452 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:42.712448    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:42.712695 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:42.712493    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:42.712695 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:42.712612    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.712586912 +0000 UTC m=+73.714833241 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : configmap references non-existent config key: service-ca.crt
Sep 10 20:59:42.712695 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:42.712630    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:59:42.712695 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:42.712644    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78d9ff86dd-646b7: secret "image-registry-tls" not found
Sep 10 20:59:42.712695 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:42.712694    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls podName:7ac41557-4731-47b3-93d2-f98b5b67e55f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.712677744 +0000 UTC m=+73.714924083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls") pod "image-registry-78d9ff86dd-646b7" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f") : secret "image-registry-tls" not found
Sep 10 20:59:42.712952 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:42.712764    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:59:42.712952 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:42.712802    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.712790858 +0000 UTC m=+73.715037183 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : secret "router-metrics-certs-default" not found
Sep 10 20:59:42.893315 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:42.893242    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7" event={"ID":"26a53a6f-5cc0-4659-ad6d-cc250ea634af","Type":"ContainerStarted","Data":"27a796fa51245a3db51c268cf710fd5868a3ec3b6a41678a6136ddf1d7d6b0da"}
Sep 10 20:59:42.910354 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:42.910293    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7" podStartSLOduration=2.415365342 podStartE2EDuration="4.910275829s" totalImagesPullingTime="2.494910487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:42.910231841 +0000 UTC m=+69.912478190" watchObservedRunningTime="2026-09-10 20:59:42.910275829 +0000 UTC m=+69.912522175"
Sep 10 20:59:44.900240 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:44.900207    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kd8gq" event={"ID":"784ee7bb-d459-4694-b60e-10ef820b238a","Type":"ContainerStarted","Data":"3ca3fa7a044b0ca4a09591147dbdae61292a31dec9317220f4ea54fd051b313d"}
Sep 10 20:59:44.900645 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:44.900319    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 20:59:44.901868 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:44.901838    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ks95v" event={"ID":"0cf2fb43-34ef-4996-b746-31482ad247ab","Type":"ContainerStarted","Data":"86a9eefac054db94de59d6cb6f46c7f145ebdf27c128c4d8f80af32fe981032a"}
Sep 10 20:59:44.923238 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:44.923189    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kd8gq" podStartSLOduration=66.969463572 podStartE2EDuration="1m11.923176691s" totalImagesPullingTime="4.953713119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:44.922676309 +0000 UTC m=+71.924922656" watchObservedRunningTime="2026-09-10 20:59:44.923176691 +0000 UTC m=+71.925423028"
Sep 10 20:59:44.940420 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:44.940369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ks95v" podStartSLOduration=66.952572888 podStartE2EDuration="1m11.940354853s" totalImagesPullingTime="4.987781965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 20:59:44.940254779 +0000 UTC m=+71.942501127" watchObservedRunningTime="2026-09-10 20:59:44.940354853 +0000 UTC m=+71.942601201"
Sep 10 20:59:45.713296 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.713255    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dprcj"]
Sep 10 20:59:45.716905 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.716882    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.719291 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.719262    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 20:59:45.719423 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.719292    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 20:59:45.719423 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.719332    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89wjp\""
Sep 10 20:59:45.719423 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.719374    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 20:59:45.719866 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.719853    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 20:59:45.726441 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.726421    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dprcj"]
Sep 10 20:59:45.740449 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.740420    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d024674-eb1f-40e7-960c-d7d8ea66eb75-data-volume\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.740589 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.740491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d024674-eb1f-40e7-960c-d7d8ea66eb75-crio-socket\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.740589 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.740572    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.740701 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.740605    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.740701 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.740649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cr8g\" (UniqueName: \"kubernetes.io/projected/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-api-access-2cr8g\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.841701 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.841665    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d024674-eb1f-40e7-960c-d7d8ea66eb75-data-volume\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.841902 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.841769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d024674-eb1f-40e7-960c-d7d8ea66eb75-crio-socket\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.841902 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.841806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.841902 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.841830    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.841902 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.841865    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cr8g\" (UniqueName: \"kubernetes.io/projected/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-api-access-2cr8g\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.842105 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:45.841904    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:45.842105 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:45.841967    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.34195047 +0000 UTC m=+73.344196794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:45.842105 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.841905    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1d024674-eb1f-40e7-960c-d7d8ea66eb75-crio-socket\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.842105 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:45.841983    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:45.842105 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:45.842054    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:46.342035127 +0000 UTC m=+73.344281468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:45.842105 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.842057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1d024674-eb1f-40e7-960c-d7d8ea66eb75-data-volume\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:45.852188 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:45.852155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cr8g\" (UniqueName: \"kubernetes.io/projected/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-api-access-2cr8g\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:46.346689 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:46.346642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:46.346689 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:46.346684    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:46.347100 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.346766    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:46.347100 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.346832    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:46.347100 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.346836    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:47.346819589 +0000 UTC m=+74.349065914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:46.347100 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.346887    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:47.346875852 +0000 UTC m=+74.349122177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:46.749577 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:46.749540    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:46.749577 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:46.749582    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:46.749774 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:46.749625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:46.749774 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.749719    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:59:46.749774 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.749762    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:59:46.749774 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.749728    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:54.749706029 +0000 UTC m=+81.751952372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : configmap references non-existent config key: service-ca.crt
Sep 10 20:59:46.749774 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.749776    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78d9ff86dd-646b7: secret "image-registry-tls" not found
Sep 10 20:59:46.749936 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.749799    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 20:59:54.749779193 +0000 UTC m=+81.752025524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : secret "router-metrics-certs-default" not found
Sep 10 20:59:46.749936 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:46.749820    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls podName:7ac41557-4731-47b3-93d2-f98b5b67e55f nodeName:}" failed. No retries permitted until 2026-09-10 20:59:54.749808152 +0000 UTC m=+81.752054481 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls") pod "image-registry-78d9ff86dd-646b7" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f") : secret "image-registry-tls" not found
Sep 10 20:59:47.354110 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:47.354067    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:47.354110 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:47.354107    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:47.354564 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:47.354200    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:47.354564 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:47.354238    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:47.354564 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:47.354273    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:49.35425393 +0000 UTC m=+76.356500254 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:47.354564 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:47.354291    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:49.354279704 +0000 UTC m=+76.356526028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:48.274877 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:48.274848    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gp8hn_7ff24c61-0ba5-4a58-a41c-95284c27b8d3/dns-node-resolver/0.log"
Sep 10 20:59:49.369074 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:49.369034    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:49.369074 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:49.369070    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:49.369498 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:49.369179    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:49.369498 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:49.369192    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:49.369498 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:49.369249    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:53.369234005 +0000 UTC m=+80.371480331 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:49.369498 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:49.369261    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 20:59:53.369254971 +0000 UTC m=+80.371501296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:49.472984 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:49.472961    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctnsl_372eb974-c525-4f33-954a-23c699ac2016/node-ca/0.log"
Sep 10 20:59:50.273207 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:50.273178    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fwkbp_2ba3f4ef-5e26-47d9-92b0-37bae0dd5650/serve-healthcheck-canary/0.log"
Sep 10 20:59:51.075368 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:51.075326    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-4f6l7_26a53a6f-5cc0-4659-ad6d-cc250ea634af/kube-storage-version-migrator-operator/0.log"
Sep 10 20:59:53.405940 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:53.405900    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:53.405940 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:53.405940    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 20:59:53.406411 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:53.406027    2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 20:59:53.406411 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:53.406057    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:53.406411 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:53.406100    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:01.4060857 +0000 UTC m=+88.408332024 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : secret "insights-runtime-extractor-tls" not found
Sep 10 20:59:53.406411 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:53.406113    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:01.406106526 +0000 UTC m=+88.408352851 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : configmap "kube-rbac-proxy" not found
Sep 10 20:59:54.816103 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:54.815932    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:54.816103 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:54.815986    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 20:59:54.816103 ip-10-0-128-236 kubenswrapper[2580]: I0910 20:59:54.816021    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls\") pod \"image-registry-78d9ff86dd-646b7\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") " pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 20:59:54.816595 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:54.816148    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 21:00:10.8161047 +0000 UTC m=+97.818351050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : configmap references non-existent config key: service-ca.crt
Sep 10 20:59:54.816595 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:54.816163    2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 20:59:54.816595 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:54.816174    2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-78d9ff86dd-646b7: secret "image-registry-tls" not found
Sep 10 20:59:54.816595 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:54.816198    2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 10 20:59:54.816595 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:54.816217    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls podName:7ac41557-4731-47b3-93d2-f98b5b67e55f nodeName:}" failed. No retries permitted until 2026-09-10 21:00:10.816199935 +0000 UTC m=+97.818446263 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls") pod "image-registry-78d9ff86dd-646b7" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f") : secret "image-registry-tls" not found
Sep 10 20:59:54.816595 ip-10-0-128-236 kubenswrapper[2580]: E0910 20:59:54.816271    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs podName:ee33f0ef-90c8-4e2c-b34d-0459c159036d nodeName:}" failed. No retries permitted until 2026-09-10 21:00:10.816253609 +0000 UTC m=+97.818499949 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs") pod "router-default-7bb47ccbbb-cpmj2" (UID: "ee33f0ef-90c8-4e2c-b34d-0459c159036d") : secret "router-metrics-certs-default" not found
Sep 10 21:00:01.465812 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:01.465769    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 21:00:01.465812 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:01.465810    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 21:00:01.466317 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:01.465918    2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 21:00:01.466317 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:01.465974    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls podName:1d024674-eb1f-40e7-960c-d7d8ea66eb75 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:17.465959471 +0000 UTC m=+104.468205796 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dprcj" (UID: "1d024674-eb1f-40e7-960c-d7d8ea66eb75") : secret "insights-runtime-extractor-tls" not found
Sep 10 21:00:01.466397 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:01.466316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1d024674-eb1f-40e7-960c-d7d8ea66eb75-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 21:00:08.004803 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:08.004766    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78d9ff86dd-646b7"]
Sep 10 21:00:08.005208 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:08.004985    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-78d9ff86dd-646b7" podUID="7ac41557-4731-47b3-93d2-f98b5b67e55f"
Sep 10 21:00:08.964346 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:08.964315    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 21:00:08.972286 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:08.972254    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 21:00:09.123207 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123161    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-bound-sa-token\") pod \"7ac41557-4731-47b3-93d2-f98b5b67e55f\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") "
Sep 10 21:00:09.123655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123224    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-installation-pull-secrets\") pod \"7ac41557-4731-47b3-93d2-f98b5b67e55f\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") "
Sep 10 21:00:09.123655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123251    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-certificates\") pod \"7ac41557-4731-47b3-93d2-f98b5b67e55f\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") "
Sep 10 21:00:09.123655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123272    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-image-registry-private-configuration\") pod \"7ac41557-4731-47b3-93d2-f98b5b67e55f\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") "
Sep 10 21:00:09.123655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123296    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ac41557-4731-47b3-93d2-f98b5b67e55f-ca-trust-extracted\") pod \"7ac41557-4731-47b3-93d2-f98b5b67e55f\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") "
Sep 10 21:00:09.123655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123323    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c88t2\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-kube-api-access-c88t2\") pod \"7ac41557-4731-47b3-93d2-f98b5b67e55f\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") "
Sep 10 21:00:09.123655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-trusted-ca\") pod \"7ac41557-4731-47b3-93d2-f98b5b67e55f\" (UID: \"7ac41557-4731-47b3-93d2-f98b5b67e55f\") "
Sep 10 21:00:09.123655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123629    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7ac41557-4731-47b3-93d2-f98b5b67e55f" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:09.123948 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ac41557-4731-47b3-93d2-f98b5b67e55f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7ac41557-4731-47b3-93d2-f98b5b67e55f" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:00:09.123948 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.123850    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7ac41557-4731-47b3-93d2-f98b5b67e55f" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:00:09.125800 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.125770    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7ac41557-4731-47b3-93d2-f98b5b67e55f" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:09.125909 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.125796    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7ac41557-4731-47b3-93d2-f98b5b67e55f" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:09.126152 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.126096    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7ac41557-4731-47b3-93d2-f98b5b67e55f" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:00:09.126152 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.126093    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-kube-api-access-c88t2" (OuterVolumeSpecName: "kube-api-access-c88t2") pod "7ac41557-4731-47b3-93d2-f98b5b67e55f" (UID: "7ac41557-4731-47b3-93d2-f98b5b67e55f"). InnerVolumeSpecName "kube-api-access-c88t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:00:09.224109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.224018    2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-installation-pull-secrets\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:00:09.224109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.224048    2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-certificates\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:00:09.224109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.224057    2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7ac41557-4731-47b3-93d2-f98b5b67e55f-image-registry-private-configuration\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:00:09.224109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.224070    2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7ac41557-4731-47b3-93d2-f98b5b67e55f-ca-trust-extracted\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:00:09.224109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.224078    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c88t2\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-kube-api-access-c88t2\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:00:09.224109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.224088    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7ac41557-4731-47b3-93d2-f98b5b67e55f-trusted-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:00:09.224109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.224097    2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-bound-sa-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:00:09.966743 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:09.966712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-78d9ff86dd-646b7"
Sep 10 21:00:10.005099 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:10.005070    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-78d9ff86dd-646b7"]
Sep 10 21:00:10.008846 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:10.008822    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-78d9ff86dd-646b7"]
Sep 10 21:00:10.130065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:10.130017    2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7ac41557-4731-47b3-93d2-f98b5b67e55f-registry-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:00:10.835216 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:10.835159    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:10.835216 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:10.835218    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:10.835911 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:10.835871    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ee33f0ef-90c8-4e2c-b34d-0459c159036d-service-ca-bundle\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:10.837690 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:10.837672    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee33f0ef-90c8-4e2c-b34d-0459c159036d-metrics-certs\") pod \"router-default-7bb47ccbbb-cpmj2\" (UID: \"ee33f0ef-90c8-4e2c-b34d-0459c159036d\") " pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:11.080926 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:11.080885    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:11.360163 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:11.360070    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7bb47ccbbb-cpmj2"]
Sep 10 21:00:11.363360 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:11.363333    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee33f0ef_90c8_4e2c_b34d_0459c159036d.slice/crio-655be8b0ef16bc126bb1d12057b5f6c47416694bc17c222c0508572f0f6ef3eb WatchSource:0}: Error finding container 655be8b0ef16bc126bb1d12057b5f6c47416694bc17c222c0508572f0f6ef3eb: Status 404 returned error can't find the container with id 655be8b0ef16bc126bb1d12057b5f6c47416694bc17c222c0508572f0f6ef3eb
Sep 10 21:00:11.516795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:11.516760    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ac41557-4731-47b3-93d2-f98b5b67e55f" path="/var/lib/kubelet/pods/7ac41557-4731-47b3-93d2-f98b5b67e55f/volumes"
Sep 10 21:00:11.973609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:11.973574    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2" event={"ID":"ee33f0ef-90c8-4e2c-b34d-0459c159036d","Type":"ContainerStarted","Data":"fece18854b8cca7a95b9dca933715ee93f318b7e69cf84221b04d0a49ccdf12d"}
Sep 10 21:00:11.973858 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:11.973622    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2" event={"ID":"ee33f0ef-90c8-4e2c-b34d-0459c159036d","Type":"ContainerStarted","Data":"655be8b0ef16bc126bb1d12057b5f6c47416694bc17c222c0508572f0f6ef3eb"}
Sep 10 21:00:11.995542 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:11.995492    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2" podStartSLOduration=33.995476196 podStartE2EDuration="33.995476196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:11.994461362 +0000 UTC m=+98.996707709" watchObservedRunningTime="2026-09-10 21:00:11.995476196 +0000 UTC m=+98.997722545"
Sep 10 21:00:12.081352 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:12.081316    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:12.083804 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:12.083782    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:12.751248 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:12.751205    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 21:00:12.753606 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:12.753588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/107d30b3-f5f5-455b-8834-509a7b83772f-metrics-tls\") pod \"dns-default-l68d2\" (UID: \"107d30b3-f5f5-455b-8834-509a7b83772f\") " pod="openshift-dns/dns-default-l68d2"
Sep 10 21:00:12.830622 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:12.830586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7j9n7\""
Sep 10 21:00:12.838365 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:12.838341    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-l68d2"
Sep 10 21:00:12.977291 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:12.977252    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:12.978546 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:12.978528    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7bb47ccbbb-cpmj2"
Sep 10 21:00:13.113544 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:13.113514    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-l68d2"]
Sep 10 21:00:13.116701 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:13.116676    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107d30b3_f5f5_455b_8834_509a7b83772f.slice/crio-1a79078f3003d6e0a9dbb36d1b99944aa6d5004d583f8d0f510d0677c0f5faba WatchSource:0}: Error finding container 1a79078f3003d6e0a9dbb36d1b99944aa6d5004d583f8d0f510d0677c0f5faba: Status 404 returned error can't find the container with id 1a79078f3003d6e0a9dbb36d1b99944aa6d5004d583f8d0f510d0677c0f5faba
Sep 10 21:00:13.981857 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:13.981813    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l68d2" event={"ID":"107d30b3-f5f5-455b-8834-509a7b83772f","Type":"ContainerStarted","Data":"1a79078f3003d6e0a9dbb36d1b99944aa6d5004d583f8d0f510d0677c0f5faba"}
Sep 10 21:00:14.986794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:14.986748    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l68d2" event={"ID":"107d30b3-f5f5-455b-8834-509a7b83772f","Type":"ContainerStarted","Data":"cc30f9ab1ffb59e502cafadb77c7adda813cd1e518424d1838ccc1284029097b"}
Sep 10 21:00:14.987230 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:14.986815    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-l68d2" event={"ID":"107d30b3-f5f5-455b-8834-509a7b83772f","Type":"ContainerStarted","Data":"748099b76847e6100f7450ac3ce8bab299cafcc11388a99438525ad450a8fe60"}
Sep 10 21:00:15.009236 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:15.009183    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-l68d2" podStartSLOduration=65.599507305 podStartE2EDuration="1m7.009167616s" totalImagesPullingTime="1.409660311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:15.007950364 +0000 UTC m=+102.010196710" watchObservedRunningTime="2026-09-10 21:00:15.009167616 +0000 UTC m=+102.011413965"
Sep 10 21:00:15.907385 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:15.907349    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kd8gq"
Sep 10 21:00:15.989751 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:15.989723    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-l68d2"
Sep 10 21:00:17.483383 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:17.483345    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 21:00:17.485732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:17.485713    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1d024674-eb1f-40e7-960c-d7d8ea66eb75-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dprcj\" (UID: \"1d024674-eb1f-40e7-960c-d7d8ea66eb75\") " pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 21:00:17.526405 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:17.526363    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dprcj"
Sep 10 21:00:17.810707 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:17.810632    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dprcj"]
Sep 10 21:00:17.813875 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:17.813852    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d024674_eb1f_40e7_960c_d7d8ea66eb75.slice/crio-2d784f5242ad1af9ae3ad398e16eea4542e1a169dc554cf3f39b1dcef076ae8e WatchSource:0}: Error finding container 2d784f5242ad1af9ae3ad398e16eea4542e1a169dc554cf3f39b1dcef076ae8e: Status 404 returned error can't find the container with id 2d784f5242ad1af9ae3ad398e16eea4542e1a169dc554cf3f39b1dcef076ae8e
Sep 10 21:00:17.996810 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:17.996771    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dprcj" event={"ID":"1d024674-eb1f-40e7-960c-d7d8ea66eb75","Type":"ContainerStarted","Data":"28b479e6cc78984973660e6acc837c62df7ff7b089e37b501639279ce10719cc"}
Sep 10 21:00:17.996972 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:17.996831    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dprcj" event={"ID":"1d024674-eb1f-40e7-960c-d7d8ea66eb75","Type":"ContainerStarted","Data":"2d784f5242ad1af9ae3ad398e16eea4542e1a169dc554cf3f39b1dcef076ae8e"}
Sep 10 21:00:20.005261 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:20.005209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dprcj" event={"ID":"1d024674-eb1f-40e7-960c-d7d8ea66eb75","Type":"ContainerStarted","Data":"6b333ffd930e2bff21f720af1cef292fd14ca5f1ad805380902992b742f39752"}
Sep 10 21:00:21.592469 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.592436    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68b58fd74f-7vv5p"]
Sep 10 21:00:21.594496 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.594480    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.599733 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.599709    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7tkmb\""
Sep 10 21:00:21.600008 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.599990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 21:00:21.600522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.600411    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 21:00:21.600522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.600420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 21:00:21.600522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.600438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 21:00:21.600522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.600468    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 21:00:21.600737 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.600420    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 21:00:21.600737 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.600719    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 21:00:21.608770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.608746    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 21:00:21.611365 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.611341    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b58fd74f-7vv5p"]
Sep 10 21:00:21.712458 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.712413    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-service-ca\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.712458 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.712465    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-config\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.712712 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.712499    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-oauth-serving-cert\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.712712 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.712552    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-trusted-ca-bundle\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.712712 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.712579    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-oauth-config\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.712712 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.712608    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8cdz\" (UniqueName: \"kubernetes.io/projected/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-kube-api-access-r8cdz\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.712712 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.712632    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-serving-cert\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.813791 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.813756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-trusted-ca-bundle\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.813791 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.813790    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-oauth-config\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.813947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.813812    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8cdz\" (UniqueName: \"kubernetes.io/projected/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-kube-api-access-r8cdz\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.813947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.813829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-serving-cert\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.814037 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.813953    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-service-ca\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.814037 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.814011    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-config\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.814160 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.814043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-oauth-serving-cert\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.814757 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.814727    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-service-ca\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.814854 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.814802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-trusted-ca-bundle\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.814854 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.814802    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-oauth-serving-cert\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.814994 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.814969    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-config\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.816479 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.816438    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-serving-cert\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.816479 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.816469    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-oauth-config\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.823430 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.823409    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8cdz\" (UniqueName: \"kubernetes.io/projected/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-kube-api-access-r8cdz\") pod \"console-68b58fd74f-7vv5p\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") " pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:21.903787 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:21.903747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:22.012994 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:22.012959    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dprcj" event={"ID":"1d024674-eb1f-40e7-960c-d7d8ea66eb75","Type":"ContainerStarted","Data":"7bffde97f70ffef3827146660444d39bf882c8c0c2e2a1d0c3e91068042d2004"}
Sep 10 21:00:22.034667 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:22.034605    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dprcj" podStartSLOduration=34.122368446 podStartE2EDuration="37.03458314s" totalImagesPullingTime="2.912214694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 20:59:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:22.033439173 +0000 UTC m=+109.035685519" watchObservedRunningTime="2026-09-10 21:00:22.03458314 +0000 UTC m=+109.036829490"
Sep 10 21:00:22.184742 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:22.184648    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68b58fd74f-7vv5p"]
Sep 10 21:00:22.188517 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:22.188491    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e76e1be_1cff_49e7_96a2_1aa9af8067e7.slice/crio-3f6fd1bc33f6301565eccf53f320386668ff574bc4304abfc5a510c7a8d4560b WatchSource:0}: Error finding container 3f6fd1bc33f6301565eccf53f320386668ff574bc4304abfc5a510c7a8d4560b: Status 404 returned error can't find the container with id 3f6fd1bc33f6301565eccf53f320386668ff574bc4304abfc5a510c7a8d4560b
Sep 10 21:00:23.018543 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:23.018502    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b58fd74f-7vv5p" event={"ID":"6e76e1be-1cff-49e7-96a2-1aa9af8067e7","Type":"ContainerStarted","Data":"3f6fd1bc33f6301565eccf53f320386668ff574bc4304abfc5a510c7a8d4560b"}
Sep 10 21:00:25.994887 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:25.994856    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-l68d2"
Sep 10 21:00:26.029820 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:26.029783    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b58fd74f-7vv5p" event={"ID":"6e76e1be-1cff-49e7-96a2-1aa9af8067e7","Type":"ContainerStarted","Data":"af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311"}
Sep 10 21:00:26.052331 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:26.052282    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68b58fd74f-7vv5p" podStartSLOduration=2.226139701 podStartE2EDuration="5.052264646s" totalImagesPullingTime="2.826124945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:26.051423734 +0000 UTC m=+113.053670081" watchObservedRunningTime="2026-09-10 21:00:26.052264646 +0000 UTC m=+113.054511048"
Sep 10 21:00:27.441346 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.441313    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"]
Sep 10 21:00:27.443438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.443422    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.445989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.445966    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 10 21:00:27.445989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.445974    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 21:00:27.446205 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.445999    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 10 21:00:27.446660 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.446639    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 21:00:27.446734 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.446686    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-pgkck\""
Sep 10 21:00:27.446734 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.446693    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 21:00:27.457250 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.457229    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"]
Sep 10 21:00:27.561791 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.561759    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8992\" (UniqueName: \"kubernetes.io/projected/be2b2bed-d84a-440b-ace3-c746319c2dd0-kube-api-access-f8992\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.561791 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.561796    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be2b2bed-d84a-440b-ace3-c746319c2dd0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.561982 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.561815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be2b2bed-d84a-440b-ace3-c746319c2dd0-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.561982 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.561839    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/be2b2bed-d84a-440b-ace3-c746319c2dd0-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.662451 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.662404    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8992\" (UniqueName: \"kubernetes.io/projected/be2b2bed-d84a-440b-ace3-c746319c2dd0-kube-api-access-f8992\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.662625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.662462    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be2b2bed-d84a-440b-ace3-c746319c2dd0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.662625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.662496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be2b2bed-d84a-440b-ace3-c746319c2dd0-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.662625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.662527    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/be2b2bed-d84a-440b-ace3-c746319c2dd0-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.663337 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.663309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be2b2bed-d84a-440b-ace3-c746319c2dd0-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.665143 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.665106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/be2b2bed-d84a-440b-ace3-c746319c2dd0-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.665143 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.665112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/be2b2bed-d84a-440b-ace3-c746319c2dd0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.671600 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.671579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8992\" (UniqueName: \"kubernetes.io/projected/be2b2bed-d84a-440b-ace3-c746319c2dd0-kube-api-access-f8992\") pod \"prometheus-operator-68bcd7f987-49rkn\" (UID: \"be2b2bed-d84a-440b-ace3-c746319c2dd0\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:27.752305 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:27.752225    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"
Sep 10 21:00:28.022147 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:28.022052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-49rkn"]
Sep 10 21:00:28.025034 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:28.025011    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2b2bed_d84a_440b_ace3_c746319c2dd0.slice/crio-0cb6949bc44ede66f1e3a481fd4f648181c6293f318dc87a3bdb240b8409d71f WatchSource:0}: Error finding container 0cb6949bc44ede66f1e3a481fd4f648181c6293f318dc87a3bdb240b8409d71f: Status 404 returned error can't find the container with id 0cb6949bc44ede66f1e3a481fd4f648181c6293f318dc87a3bdb240b8409d71f
Sep 10 21:00:28.037395 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:28.037364    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn" event={"ID":"be2b2bed-d84a-440b-ace3-c746319c2dd0","Type":"ContainerStarted","Data":"0cb6949bc44ede66f1e3a481fd4f648181c6293f318dc87a3bdb240b8409d71f"}
Sep 10 21:00:30.045861 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:30.045822    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn" event={"ID":"be2b2bed-d84a-440b-ace3-c746319c2dd0","Type":"ContainerStarted","Data":"68ca9bc8fdd02fcd8952b06826eff8a4c5d69d48155cd0cdfdbbf9491e0a5e7e"}
Sep 10 21:00:30.046373 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:30.045880    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn" event={"ID":"be2b2bed-d84a-440b-ace3-c746319c2dd0","Type":"ContainerStarted","Data":"c867ba0a363b7009a12abac4fe6557924ad2e68f6f72e8bec17757a419480053"}
Sep 10 21:00:30.063879 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:30.063821    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-49rkn" podStartSLOduration=1.640448498 podStartE2EDuration="3.063803036s" totalImagesPullingTime="1.423354538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:30.063154816 +0000 UTC m=+117.065401157" watchObservedRunningTime="2026-09-10 21:00:30.063803036 +0000 UTC m=+117.066049385"
Sep 10 21:00:31.850867 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.850831    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"]
Sep 10 21:00:31.853053 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.853037    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:31.855640 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.855614    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zqfm8\""
Sep 10 21:00:31.855787 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.855656    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 21:00:31.855998 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.855983    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 21:00:31.870028 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.869993    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"]
Sep 10 21:00:31.871149 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.871111    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-gwdzd"]
Sep 10 21:00:31.873611 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.873591    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.875950 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.875934    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 21:00:31.876054 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.875982    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 21:00:31.876305 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.876273    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 21:00:31.876436 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.876312    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rpll2\""
Sep 10 21:00:31.903978 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.903945    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:31.904162 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.903999    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:31.912780 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.912754    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:31.996357 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996328    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-root\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996562 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996369    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-textfile\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996562 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996389    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fc77\" (UniqueName: \"kubernetes.io/projected/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-kube-api-access-6fc77\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996562 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996419    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996562 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996474    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-metrics-client-ca\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996562 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996535    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5492f628-1e3d-4399-9e78-e509ebd03d25-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:31.996794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996564    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:31.996794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996603    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996662    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-sys\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996679    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ql7\" (UniqueName: \"kubernetes.io/projected/5492f628-1e3d-4399-9e78-e509ebd03d25-kube-api-access-r4ql7\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:31.996794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996694    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-wtmp\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:31.996794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:31.996731    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.056937 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.056910    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:00:32.097968 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.097933    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5492f628-1e3d-4399-9e78-e509ebd03d25-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.098209 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.097978    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.098209 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.097998    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098209 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:32.098088    2580 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 10 21:00:32.098209 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098114    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098209 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:32.098188    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-tls podName:5492f628-1e3d-4399-9e78-e509ebd03d25 nodeName:}" failed. No retries permitted until 2026-09-10 21:00:32.598165325 +0000 UTC m=+119.600411651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-tls") pod "openshift-state-metrics-67dcf8c979-ks5dc" (UID: "5492f628-1e3d-4399-9e78-e509ebd03d25") : secret "openshift-state-metrics-tls" not found
Sep 10 21:00:32.098209 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098212    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-sys\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098574 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098236    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ql7\" (UniqueName: \"kubernetes.io/projected/5492f628-1e3d-4399-9e78-e509ebd03d25-kube-api-access-r4ql7\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.098574 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:32.098248    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 21:00:32.098574 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098263    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-wtmp\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098574 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:32.098328    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls podName:19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef nodeName:}" failed. No retries permitted until 2026-09-10 21:00:32.598310025 +0000 UTC m=+119.600556365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls") pod "node-exporter-gwdzd" (UID: "19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef") : secret "node-exporter-tls" not found
Sep 10 21:00:32.098574 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098364    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-sys\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098574 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-wtmp\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098877 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098354    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.098877 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098661    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-root\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098877 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098719    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5492f628-1e3d-4399-9e78-e509ebd03d25-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.098877 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098724    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-textfile\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098877 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098780    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fc77\" (UniqueName: \"kubernetes.io/projected/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-kube-api-access-6fc77\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.098877 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.099217 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-metrics-client-ca\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.099217 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098977    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-textfile\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.099217 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.098995    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-accelerators-collector-config\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.099377 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.099282    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-root\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.099618 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.099597    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-metrics-client-ca\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.101023 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.100974    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.101210 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.101191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.127645 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.127609    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fc77\" (UniqueName: \"kubernetes.io/projected/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-kube-api-access-6fc77\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.127791 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.127712    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ql7\" (UniqueName: \"kubernetes.io/projected/5492f628-1e3d-4399-9e78-e509ebd03d25-kube-api-access-r4ql7\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.604277 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.604237    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.604277 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.604276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:32.604494 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:32.604395    2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 21:00:32.604494 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:00:32.604474    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls podName:19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef nodeName:}" failed. No retries permitted until 2026-09-10 21:00:33.604456279 +0000 UTC m=+120.606702621 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls") pod "node-exporter-gwdzd" (UID: "19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef") : secret "node-exporter-tls" not found
Sep 10 21:00:32.606873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.606849    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5492f628-1e3d-4399-9e78-e509ebd03d25-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-ks5dc\" (UID: \"5492f628-1e3d-4399-9e78-e509ebd03d25\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:32.762196 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:32.762157    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"
Sep 10 21:00:33.052013 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:33.051988    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc"]
Sep 10 21:00:33.053685 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:33.053660    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5492f628_1e3d_4399_9e78_e509ebd03d25.slice/crio-2907cad641139d4d8ed8b282c85db793301c8fff15e92f5c50a6bae58fce43e4 WatchSource:0}: Error finding container 2907cad641139d4d8ed8b282c85db793301c8fff15e92f5c50a6bae58fce43e4: Status 404 returned error can't find the container with id 2907cad641139d4d8ed8b282c85db793301c8fff15e92f5c50a6bae58fce43e4
Sep 10 21:00:33.614195 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:33.614157    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:33.616469 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:33.616444    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef-node-exporter-tls\") pod \"node-exporter-gwdzd\" (UID: \"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef\") " pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:33.684826 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:33.684793    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rpll2\""
Sep 10 21:00:33.692824 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:33.692795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-gwdzd"
Sep 10 21:00:33.704504 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:33.704308    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19a6ea4d_d3fd_4c0e_a687_ee0bae8a60ef.slice/crio-b1afd45d2b6dcd31de22907f55214937312126df68df353be79a892cc9f50957 WatchSource:0}: Error finding container b1afd45d2b6dcd31de22907f55214937312126df68df353be79a892cc9f50957: Status 404 returned error can't find the container with id b1afd45d2b6dcd31de22907f55214937312126df68df353be79a892cc9f50957
Sep 10 21:00:34.062364 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:34.062305    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc" event={"ID":"5492f628-1e3d-4399-9e78-e509ebd03d25","Type":"ContainerStarted","Data":"50678af5a4820f5bdabee7436500b63e7cf7733bb4c08c03693358891e72e489"}
Sep 10 21:00:34.062845 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:34.062409    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc" event={"ID":"5492f628-1e3d-4399-9e78-e509ebd03d25","Type":"ContainerStarted","Data":"79fe2c19f6123ab27feb90c7c7af009ec4c16dfd26709e04bb225429a3656cd8"}
Sep 10 21:00:34.062845 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:34.062440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc" event={"ID":"5492f628-1e3d-4399-9e78-e509ebd03d25","Type":"ContainerStarted","Data":"2907cad641139d4d8ed8b282c85db793301c8fff15e92f5c50a6bae58fce43e4"}
Sep 10 21:00:34.063662 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:34.063635    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwdzd" event={"ID":"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef","Type":"ContainerStarted","Data":"b1afd45d2b6dcd31de22907f55214937312126df68df353be79a892cc9f50957"}
Sep 10 21:00:35.069445 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:35.069360    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc" event={"ID":"5492f628-1e3d-4399-9e78-e509ebd03d25","Type":"ContainerStarted","Data":"f8e4c832cfd340760adb03c3e9018917daebe4cc1f80235aa4e4a60a132830b6"}
Sep 10 21:00:35.071004 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:35.070978    2580 generic.go:356] "Generic (PLEG): container finished" podID="19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef" containerID="3f625123af9b8a858bf910a232965377911b01f9d8e6598f955888f9517a0084" exitCode=0
Sep 10 21:00:35.071148 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:35.071060    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwdzd" event={"ID":"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef","Type":"ContainerDied","Data":"3f625123af9b8a858bf910a232965377911b01f9d8e6598f955888f9517a0084"}
Sep 10 21:00:35.097500 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:35.097449    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-ks5dc" podStartSLOduration=3.142602598 podStartE2EDuration="4.09743375s" totalImagesPullingTime="954.831152ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:35.096839079 +0000 UTC m=+122.099085427" watchObservedRunningTime="2026-09-10 21:00:35.09743375 +0000 UTC m=+122.099680097"
Sep 10 21:00:36.076193 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.076148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwdzd" event={"ID":"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef","Type":"ContainerStarted","Data":"e2025459f3bfbf8566631c0ba14a3f2716dff7173d0404eb8bed1c2b61ee637d"}
Sep 10 21:00:36.076609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.076232    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-gwdzd" event={"ID":"19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef","Type":"ContainerStarted","Data":"2f6b64356bc56f63a44669999b5f6d77b915f88a5b5ad949d10456d0cd8e555e"}
Sep 10 21:00:36.105736 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.105689    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-gwdzd" podStartSLOduration=3.995813669 podStartE2EDuration="5.105674187s" totalImagesPullingTime="1.037671672s" totalInitContainerRuntime="72.188846ms" isStatefulPod=true podCreationTimestamp="2026-09-10 21:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:36.104567235 +0000 UTC m=+123.106813604" watchObservedRunningTime="2026-09-10 21:00:36.105674187 +0000 UTC m=+123.107920534"
Sep 10 21:00:36.764882 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.764846    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"]
Sep 10 21:00:36.767064 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.767047    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:36.769175 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.769157    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 21:00:36.769843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.769829    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-k6mdk\""
Sep 10 21:00:36.786599 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.786568    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"]
Sep 10 21:00:36.839201 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.839163    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj59q\" (UniqueName: \"kubernetes.io/projected/c33e975b-b71c-4b5c-af0a-936e5a809ae5-kube-api-access-fj59q\") pod \"monitoring-plugin-5d85778b9d-cnxrr\" (UID: \"c33e975b-b71c-4b5c-af0a-936e5a809ae5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:36.839380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.839257    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c33e975b-b71c-4b5c-af0a-936e5a809ae5-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cnxrr\" (UID: \"c33e975b-b71c-4b5c-af0a-936e5a809ae5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:36.939974 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.939937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c33e975b-b71c-4b5c-af0a-936e5a809ae5-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cnxrr\" (UID: \"c33e975b-b71c-4b5c-af0a-936e5a809ae5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:36.940164 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.939996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj59q\" (UniqueName: \"kubernetes.io/projected/c33e975b-b71c-4b5c-af0a-936e5a809ae5-kube-api-access-fj59q\") pod \"monitoring-plugin-5d85778b9d-cnxrr\" (UID: \"c33e975b-b71c-4b5c-af0a-936e5a809ae5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:36.942614 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.942588    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c33e975b-b71c-4b5c-af0a-936e5a809ae5-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-cnxrr\" (UID: \"c33e975b-b71c-4b5c-af0a-936e5a809ae5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:36.954235 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:36.954210    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj59q\" (UniqueName: \"kubernetes.io/projected/c33e975b-b71c-4b5c-af0a-936e5a809ae5-kube-api-access-fj59q\") pod \"monitoring-plugin-5d85778b9d-cnxrr\" (UID: \"c33e975b-b71c-4b5c-af0a-936e5a809ae5\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:37.075298 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:37.075218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:37.373050 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:37.372964    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"]
Sep 10 21:00:37.386936 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:37.386912    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc33e975b_b71c_4b5c_af0a_936e5a809ae5.slice/crio-d4bfd2587545bdedaa6f0f4c55192adde15691f16d401a44dd8a812876c94ede WatchSource:0}: Error finding container d4bfd2587545bdedaa6f0f4c55192adde15691f16d401a44dd8a812876c94ede: Status 404 returned error can't find the container with id d4bfd2587545bdedaa6f0f4c55192adde15691f16d401a44dd8a812876c94ede
Sep 10 21:00:38.083812 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:38.083778    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr" event={"ID":"c33e975b-b71c-4b5c-af0a-936e5a809ae5","Type":"ContainerStarted","Data":"d4bfd2587545bdedaa6f0f4c55192adde15691f16d401a44dd8a812876c94ede"}
Sep 10 21:00:39.089090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:39.089056    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr" event={"ID":"c33e975b-b71c-4b5c-af0a-936e5a809ae5","Type":"ContainerStarted","Data":"da5060e8a340b2bb1c73df348e81c19b467016a0e586781b60be80bcb03afd0e"}
Sep 10 21:00:39.089492 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:39.089249    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:39.093953 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:39.093932    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr"
Sep 10 21:00:39.112730 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:39.112685    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-cnxrr" podStartSLOduration=1.5318610449999999 podStartE2EDuration="3.112669441s" totalImagesPullingTime="1.580808396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:00:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:39.112316387 +0000 UTC m=+126.114562737" watchObservedRunningTime="2026-09-10 21:00:39.112669441 +0000 UTC m=+126.114915787"
Sep 10 21:00:43.287961 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:43.287920    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 21:00:43.290418 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:43.290382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9341a00d-f1e0-4798-b687-8f058c4e2d59-metrics-certs\") pod \"network-metrics-daemon-8wqmx\" (UID: \"9341a00d-f1e0-4798-b687-8f058c4e2d59\") " pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 21:00:43.533538 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:43.533504    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5khf6\""
Sep 10 21:00:43.542180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:43.542090    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8wqmx"
Sep 10 21:00:43.818296 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:43.818206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8wqmx"]
Sep 10 21:00:43.821135 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:00:43.821093    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9341a00d_f1e0_4798_b687_8f058c4e2d59.slice/crio-ae99375e53ffc46eb47fe89063a849bb8848891de7aa6bc3dfa37069af013e3d WatchSource:0}: Error finding container ae99375e53ffc46eb47fe89063a849bb8848891de7aa6bc3dfa37069af013e3d: Status 404 returned error can't find the container with id ae99375e53ffc46eb47fe89063a849bb8848891de7aa6bc3dfa37069af013e3d
Sep 10 21:00:44.112566 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:44.112481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8wqmx" event={"ID":"9341a00d-f1e0-4798-b687-8f058c4e2d59","Type":"ContainerStarted","Data":"ae99375e53ffc46eb47fe89063a849bb8848891de7aa6bc3dfa37069af013e3d"}
Sep 10 21:00:45.117789 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:45.117679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8wqmx" event={"ID":"9341a00d-f1e0-4798-b687-8f058c4e2d59","Type":"ContainerStarted","Data":"b5768d6a442e7e5f867cc4d975b4c0b24dccd89ef93ec988a401c8a1615f407e"}
Sep 10 21:00:45.117789 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:45.117728    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8wqmx" event={"ID":"9341a00d-f1e0-4798-b687-8f058c4e2d59","Type":"ContainerStarted","Data":"b6434086c545971836ada220c6af05f80f9c402cd27a3ba58e0aedcb4989ff53"}
Sep 10 21:00:45.141759 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:45.141709    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8wqmx" podStartSLOduration=131.146916693 podStartE2EDuration="2m12.141694202s" totalImagesPullingTime="994.777509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 20:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:00:45.140807171 +0000 UTC m=+132.143053518" watchObservedRunningTime="2026-09-10 21:00:45.141694202 +0000 UTC m=+132.143940549"
Sep 10 21:00:47.692731 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:47.692692    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b58fd74f-7vv5p"]
Sep 10 21:00:48.131074 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:48.131034    2580 generic.go:356] "Generic (PLEG): container finished" podID="26a53a6f-5cc0-4659-ad6d-cc250ea634af" containerID="27a796fa51245a3db51c268cf710fd5868a3ec3b6a41678a6136ddf1d7d6b0da" exitCode=0
Sep 10 21:00:48.131302 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:48.131107    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7" event={"ID":"26a53a6f-5cc0-4659-ad6d-cc250ea634af","Type":"ContainerDied","Data":"27a796fa51245a3db51c268cf710fd5868a3ec3b6a41678a6136ddf1d7d6b0da"}
Sep 10 21:00:48.131463 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:48.131450    2580 scope.go:117] "RemoveContainer" containerID="27a796fa51245a3db51c268cf710fd5868a3ec3b6a41678a6136ddf1d7d6b0da"
Sep 10 21:00:49.136993 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:00:49.136961    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-4f6l7" event={"ID":"26a53a6f-5cc0-4659-ad6d-cc250ea634af","Type":"ContainerStarted","Data":"9280e1d1d68ba6e1ae9c378ca70180b150f37b6961fd31ec3e8cb6a5f454872d"}
Sep 10 21:01:12.711617 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:12.711571    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68b58fd74f-7vv5p" podUID="6e76e1be-1cff-49e7-96a2-1aa9af8067e7" containerName="console" containerID="cri-o://af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311" gracePeriod=15
Sep 10 21:01:12.944626 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:12.944604    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:01:13.006899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.006782    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r8cdz\" (UniqueName: \"kubernetes.io/projected/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-kube-api-access-r8cdz\") pod \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") "
Sep 10 21:01:13.006899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.006833    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-service-ca\") pod \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") "
Sep 10 21:01:13.006899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.006853    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-oauth-serving-cert\") pod \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") "
Sep 10 21:01:13.006899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.006893    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-serving-cert\") pod \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") "
Sep 10 21:01:13.007270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.006918    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-trusted-ca-bundle\") pod \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") "
Sep 10 21:01:13.007270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.006933    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-oauth-config\") pod \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") "
Sep 10 21:01:13.007270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.006952    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-config\") pod \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\" (UID: \"6e76e1be-1cff-49e7-96a2-1aa9af8067e7\") "
Sep 10 21:01:13.007425 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.007370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-service-ca" (OuterVolumeSpecName: "service-ca") pod "6e76e1be-1cff-49e7-96a2-1aa9af8067e7" (UID: "6e76e1be-1cff-49e7-96a2-1aa9af8067e7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:13.007425 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.007382    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6e76e1be-1cff-49e7-96a2-1aa9af8067e7" (UID: "6e76e1be-1cff-49e7-96a2-1aa9af8067e7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:13.007425 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.007411    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-config" (OuterVolumeSpecName: "console-config") pod "6e76e1be-1cff-49e7-96a2-1aa9af8067e7" (UID: "6e76e1be-1cff-49e7-96a2-1aa9af8067e7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:13.007614 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.007434    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6e76e1be-1cff-49e7-96a2-1aa9af8067e7" (UID: "6e76e1be-1cff-49e7-96a2-1aa9af8067e7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:01:13.009270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.009245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-kube-api-access-r8cdz" (OuterVolumeSpecName: "kube-api-access-r8cdz") pod "6e76e1be-1cff-49e7-96a2-1aa9af8067e7" (UID: "6e76e1be-1cff-49e7-96a2-1aa9af8067e7"). InnerVolumeSpecName "kube-api-access-r8cdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:01:13.009707 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.009681    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6e76e1be-1cff-49e7-96a2-1aa9af8067e7" (UID: "6e76e1be-1cff-49e7-96a2-1aa9af8067e7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:01:13.009707 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.009694    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6e76e1be-1cff-49e7-96a2-1aa9af8067e7" (UID: "6e76e1be-1cff-49e7-96a2-1aa9af8067e7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:01:13.107621 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.107590    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-trusted-ca-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:01:13.107621 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.107615    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-oauth-config\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:01:13.107621 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.107624    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-config\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:01:13.107840 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.107633    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8cdz\" (UniqueName: \"kubernetes.io/projected/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-kube-api-access-r8cdz\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:01:13.107840 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.107642    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:01:13.107840 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.107650    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-oauth-serving-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:01:13.107840 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.107658    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e76e1be-1cff-49e7-96a2-1aa9af8067e7-console-serving-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:01:13.218991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.218953    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e76e1be-1cff-49e7-96a2-1aa9af8067e7" containerID="af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311" exitCode=0
Sep 10 21:01:13.219177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.219005    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68b58fd74f-7vv5p"
Sep 10 21:01:13.219177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.219018    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b58fd74f-7vv5p" event={"ID":"6e76e1be-1cff-49e7-96a2-1aa9af8067e7","Type":"ContainerDied","Data":"af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311"}
Sep 10 21:01:13.219177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.219071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68b58fd74f-7vv5p" event={"ID":"6e76e1be-1cff-49e7-96a2-1aa9af8067e7","Type":"ContainerDied","Data":"3f6fd1bc33f6301565eccf53f320386668ff574bc4304abfc5a510c7a8d4560b"}
Sep 10 21:01:13.219177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.219100    2580 scope.go:117] "RemoveContainer" containerID="af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311"
Sep 10 21:01:13.227993 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.227975    2580 scope.go:117] "RemoveContainer" containerID="af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311"
Sep 10 21:01:13.228292 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:01:13.228272    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311\": container with ID starting with af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311 not found: ID does not exist" containerID="af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311"
Sep 10 21:01:13.228365 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.228299    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311"} err="failed to get container status \"af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311\": rpc error: code = NotFound desc = could not find container \"af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311\": container with ID starting with af5a570a76d52b99f9fa1564141ad6636e6208cf8436fd6f8ac78eb3551cd311 not found: ID does not exist"
Sep 10 21:01:13.240947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.240922    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68b58fd74f-7vv5p"]
Sep 10 21:01:13.245079 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.245056    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68b58fd74f-7vv5p"]
Sep 10 21:01:13.516916 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:01:13.516879    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e76e1be-1cff-49e7-96a2-1aa9af8067e7" path="/var/lib/kubelet/pods/6e76e1be-1cff-49e7-96a2-1aa9af8067e7/volumes"
Sep 10 21:02:00.013641 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.013600    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7ddf855465-6922h"]
Sep 10 21:02:00.014843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.014811    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e76e1be-1cff-49e7-96a2-1aa9af8067e7" containerName="console"
Sep 10 21:02:00.014843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.014837    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e76e1be-1cff-49e7-96a2-1aa9af8067e7" containerName="console"
Sep 10 21:02:00.014991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.014895    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e76e1be-1cff-49e7-96a2-1aa9af8067e7" containerName="console"
Sep 10 21:02:00.017249 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.017233    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.019544 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.019512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 21:02:00.019544 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.019514    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 21:02:00.019718 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.019568    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 21:02:00.020334 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.020306    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 21:02:00.020452 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.020357    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 21:02:00.020452 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.020390    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 21:02:00.020614 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.020586    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7tkmb\""
Sep 10 21:02:00.020873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.020856    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 21:02:00.024363 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.024341    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 21:02:00.028153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.028132    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ddf855465-6922h"]
Sep 10 21:02:00.150657 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.150615    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-trusted-ca-bundle\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.150843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.150664    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-oauth-config\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.150843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.150689    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-oauth-serving-cert\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.150843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.150735    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-config\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.150843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.150758    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6ckx\" (UniqueName: \"kubernetes.io/projected/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-kube-api-access-k6ckx\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.150843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.150815    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-serving-cert\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.151031 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.150878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-service-ca\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.251422 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.251359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-trusted-ca-bundle\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.251624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.251444    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-oauth-config\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.251624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.251468    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-oauth-serving-cert\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.251624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.251504    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-config\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.251624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.251534    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6ckx\" (UniqueName: \"kubernetes.io/projected/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-kube-api-access-k6ckx\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.251624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.251589    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-serving-cert\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.251834 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.251672    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-service-ca\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.252335 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.252308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-trusted-ca-bundle\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.252335 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.252328    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-service-ca\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.252497 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.252309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-oauth-serving-cert\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.252596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.252577    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-config\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.254140 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.254090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-serving-cert\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.254239 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.254153    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-oauth-config\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.260461 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.260439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6ckx\" (UniqueName: \"kubernetes.io/projected/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-kube-api-access-k6ckx\") pod \"console-7ddf855465-6922h\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") " pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.328399 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.328303    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:00.610335 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:00.610243    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7ddf855465-6922h"]
Sep 10 21:02:01.389998 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:01.389963    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ddf855465-6922h" event={"ID":"9a1838e5-d98c-4f8a-9b10-f32467cc4aae","Type":"ContainerStarted","Data":"ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b"}
Sep 10 21:02:01.390423 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:01.390033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ddf855465-6922h" event={"ID":"9a1838e5-d98c-4f8a-9b10-f32467cc4aae","Type":"ContainerStarted","Data":"80b1dca6ecaa7e3c1f8f142512117c59d855c7cb72ba01e7307c6ec56afdc04e"}
Sep 10 21:02:01.408236 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:01.408191    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7ddf855465-6922h" podStartSLOduration=2.408177131 podStartE2EDuration="2.408177131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:02:01.40792527 +0000 UTC m=+208.410171617" watchObservedRunningTime="2026-09-10 21:02:01.408177131 +0000 UTC m=+208.410423478"
Sep 10 21:02:10.328557 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:10.328522    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:10.329016 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:10.328585    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:10.333370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:10.333347    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:10.424076 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:10.424044    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:21.034176 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:21.034138    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ddf855465-6922h"]
Sep 10 21:02:46.054002 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.053957    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7ddf855465-6922h" podUID="9a1838e5-d98c-4f8a-9b10-f32467cc4aae" containerName="console" containerID="cri-o://ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b" gracePeriod=15
Sep 10 21:02:46.294477 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.294455    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:46.380321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380297    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6ckx\" (UniqueName: \"kubernetes.io/projected/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-kube-api-access-k6ckx\") pod \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") "
Sep 10 21:02:46.380443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380326    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-serving-cert\") pod \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") "
Sep 10 21:02:46.380443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380367    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-trusted-ca-bundle\") pod \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") "
Sep 10 21:02:46.380443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380383    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-oauth-config\") pod \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") "
Sep 10 21:02:46.380443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380414    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-config\") pod \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") "
Sep 10 21:02:46.380443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380434    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-oauth-serving-cert\") pod \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") "
Sep 10 21:02:46.380678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380456    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-service-ca\") pod \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\" (UID: \"9a1838e5-d98c-4f8a-9b10-f32467cc4aae\") "
Sep 10 21:02:46.380881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380854    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-config" (OuterVolumeSpecName: "console-config") pod "9a1838e5-d98c-4f8a-9b10-f32467cc4aae" (UID: "9a1838e5-d98c-4f8a-9b10-f32467cc4aae"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:02:46.380881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380861    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9a1838e5-d98c-4f8a-9b10-f32467cc4aae" (UID: "9a1838e5-d98c-4f8a-9b10-f32467cc4aae"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:02:46.381005 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380892    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-service-ca" (OuterVolumeSpecName: "service-ca") pod "9a1838e5-d98c-4f8a-9b10-f32467cc4aae" (UID: "9a1838e5-d98c-4f8a-9b10-f32467cc4aae"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:02:46.381005 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.380889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9a1838e5-d98c-4f8a-9b10-f32467cc4aae" (UID: "9a1838e5-d98c-4f8a-9b10-f32467cc4aae"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:02:46.382689 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.382668    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9a1838e5-d98c-4f8a-9b10-f32467cc4aae" (UID: "9a1838e5-d98c-4f8a-9b10-f32467cc4aae"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:02:46.382978 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.382955    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-kube-api-access-k6ckx" (OuterVolumeSpecName: "kube-api-access-k6ckx") pod "9a1838e5-d98c-4f8a-9b10-f32467cc4aae" (UID: "9a1838e5-d98c-4f8a-9b10-f32467cc4aae"). InnerVolumeSpecName "kube-api-access-k6ckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:02:46.383068 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.382973    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9a1838e5-d98c-4f8a-9b10-f32467cc4aae" (UID: "9a1838e5-d98c-4f8a-9b10-f32467cc4aae"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:02:46.481404 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.481378    2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-config\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:02:46.481404 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.481399    2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-oauth-serving-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:02:46.481404 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.481408    2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:02:46.481582 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.481416    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6ckx\" (UniqueName: \"kubernetes.io/projected/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-kube-api-access-k6ckx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:02:46.481582 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.481426    2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-serving-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:02:46.481582 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.481434    2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-trusted-ca-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:02:46.481582 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.481444    2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a1838e5-d98c-4f8a-9b10-f32467cc4aae-console-oauth-config\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:02:46.544240 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.544204    2580 generic.go:356] "Generic (PLEG): container finished" podID="9a1838e5-d98c-4f8a-9b10-f32467cc4aae" containerID="ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b" exitCode=0
Sep 10 21:02:46.544378 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.544255    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7ddf855465-6922h"
Sep 10 21:02:46.544378 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.544287    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ddf855465-6922h" event={"ID":"9a1838e5-d98c-4f8a-9b10-f32467cc4aae","Type":"ContainerDied","Data":"ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b"}
Sep 10 21:02:46.544378 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.544333    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7ddf855465-6922h" event={"ID":"9a1838e5-d98c-4f8a-9b10-f32467cc4aae","Type":"ContainerDied","Data":"80b1dca6ecaa7e3c1f8f142512117c59d855c7cb72ba01e7307c6ec56afdc04e"}
Sep 10 21:02:46.544378 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.544352    2580 scope.go:117] "RemoveContainer" containerID="ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b"
Sep 10 21:02:46.552599 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.552583    2580 scope.go:117] "RemoveContainer" containerID="ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b"
Sep 10 21:02:46.552834 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:02:46.552820    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b\": container with ID starting with ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b not found: ID does not exist" containerID="ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b"
Sep 10 21:02:46.552878 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.552842    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b"} err="failed to get container status \"ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b\": rpc error: code = NotFound desc = could not find container \"ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b\": container with ID starting with ef9c2cb1bec677175c7c5f0e28952cf3cebcf993c376bc68697040bd46ae410b not found: ID does not exist"
Sep 10 21:02:46.563149 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.563111    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7ddf855465-6922h"]
Sep 10 21:02:46.566680 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:46.566656    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7ddf855465-6922h"]
Sep 10 21:02:47.517643 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:02:47.517604    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1838e5-d98c-4f8a-9b10-f32467cc4aae" path="/var/lib/kubelet/pods/9a1838e5-d98c-4f8a-9b10-f32467cc4aae/volumes"
Sep 10 21:03:08.758245 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.758209    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"]
Sep 10 21:03:08.758631 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.758472    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1838e5-d98c-4f8a-9b10-f32467cc4aae" containerName="console"
Sep 10 21:03:08.758631 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.758483    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1838e5-d98c-4f8a-9b10-f32467cc4aae" containerName="console"
Sep 10 21:03:08.758631 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.758523    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1838e5-d98c-4f8a-9b10-f32467cc4aae" containerName="console"
Sep 10 21:03:08.761407 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.761391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.763603 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.763584    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 21:03:08.763653 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.763626    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 21:03:08.764211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.764193    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jvspl\""
Sep 10 21:03:08.767976 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.767673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"]
Sep 10 21:03:08.838307 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.838269    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.838500 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.838326    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.838500 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.838359    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks65m\" (UniqueName: \"kubernetes.io/projected/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-kube-api-access-ks65m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.938676 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.938645    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.938876 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.938710    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.938876 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.938738    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks65m\" (UniqueName: \"kubernetes.io/projected/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-kube-api-access-ks65m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.939112 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.939090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.939193 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.939090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:08.947290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:08.947265    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks65m\" (UniqueName: \"kubernetes.io/projected/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-kube-api-access-ks65m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:09.070788 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:09.070700    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:09.340060 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:09.339977    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"]
Sep 10 21:03:09.342964 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:03:09.342922    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa24fbe8_195e_4a93_b6ac_2cc80e4b4a97.slice/crio-ccc1966a2cd49640751b80f96a586cb55d94d11e4a17fcbe52cf814891cfed88 WatchSource:0}: Error finding container ccc1966a2cd49640751b80f96a586cb55d94d11e4a17fcbe52cf814891cfed88: Status 404 returned error can't find the container with id ccc1966a2cd49640751b80f96a586cb55d94d11e4a17fcbe52cf814891cfed88
Sep 10 21:03:09.623720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:09.623689    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg" event={"ID":"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97","Type":"ContainerStarted","Data":"ccc1966a2cd49640751b80f96a586cb55d94d11e4a17fcbe52cf814891cfed88"}
Sep 10 21:03:16.651439 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:16.651402    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerID="c0574362c0c8591387cf990ec4df2ecabd092a4067474221a142c9c8a240ef53" exitCode=0
Sep 10 21:03:16.651842 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:16.651492    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg" event={"ID":"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97","Type":"ContainerDied","Data":"c0574362c0c8591387cf990ec4df2ecabd092a4067474221a142c9c8a240ef53"}
Sep 10 21:03:19.666520 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:19.666480    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerID="4b435c7bb7666fd0cb2893d62e162262ea10bb357d6dd9f7a16a9fe4e5850d4c" exitCode=0
Sep 10 21:03:19.666928 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:19.666567    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg" event={"ID":"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97","Type":"ContainerDied","Data":"4b435c7bb7666fd0cb2893d62e162262ea10bb357d6dd9f7a16a9fe4e5850d4c"}
Sep 10 21:03:22.563191 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.563150    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"]
Sep 10 21:03:22.566971 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.566949    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"
Sep 10 21:03:22.570770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.570713    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 21:03:22.570770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.570758    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-jnbtf\""
Sep 10 21:03:22.571058 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.570795    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 21:03:22.571058 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.570849    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 21:03:22.571343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.571324    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 21:03:22.576978 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.576957    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"]
Sep 10 21:03:22.602748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.602717    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"]
Sep 10 21:03:22.605449 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.605429    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.607548 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.607532    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 21:03:22.615593 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.615572    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"]
Sep 10 21:03:22.650068 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.650039    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzw87\" (UniqueName: \"kubernetes.io/projected/677e8f24-02f8-4c81-97ff-d9deeebb665c-kube-api-access-vzw87\") pod \"managed-serviceaccount-addon-agent-6c9d79d558-87fzx\" (UID: \"677e8f24-02f8-4c81-97ff-d9deeebb665c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"
Sep 10 21:03:22.650253 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.650086    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677e8f24-02f8-4c81-97ff-d9deeebb665c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c9d79d558-87fzx\" (UID: \"677e8f24-02f8-4c81-97ff-d9deeebb665c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"
Sep 10 21:03:22.750483 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.750430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c70a145-c987-41fb-8dbf-41d766d1f61d-tmp\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.750652 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.750492    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzw87\" (UniqueName: \"kubernetes.io/projected/677e8f24-02f8-4c81-97ff-d9deeebb665c-kube-api-access-vzw87\") pod \"managed-serviceaccount-addon-agent-6c9d79d558-87fzx\" (UID: \"677e8f24-02f8-4c81-97ff-d9deeebb665c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"
Sep 10 21:03:22.750652 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.750551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c70a145-c987-41fb-8dbf-41d766d1f61d-klusterlet-config\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.750652 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.750607    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjs99\" (UniqueName: \"kubernetes.io/projected/4c70a145-c987-41fb-8dbf-41d766d1f61d-kube-api-access-hjs99\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.750886 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.750688    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677e8f24-02f8-4c81-97ff-d9deeebb665c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c9d79d558-87fzx\" (UID: \"677e8f24-02f8-4c81-97ff-d9deeebb665c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"
Sep 10 21:03:22.753673 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.753641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/677e8f24-02f8-4c81-97ff-d9deeebb665c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6c9d79d558-87fzx\" (UID: \"677e8f24-02f8-4c81-97ff-d9deeebb665c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"
Sep 10 21:03:22.759347 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.759317    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzw87\" (UniqueName: \"kubernetes.io/projected/677e8f24-02f8-4c81-97ff-d9deeebb665c-kube-api-access-vzw87\") pod \"managed-serviceaccount-addon-agent-6c9d79d558-87fzx\" (UID: \"677e8f24-02f8-4c81-97ff-d9deeebb665c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"
Sep 10 21:03:22.851442 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.851359    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c70a145-c987-41fb-8dbf-41d766d1f61d-tmp\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.851442 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.851400    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c70a145-c987-41fb-8dbf-41d766d1f61d-klusterlet-config\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.851442 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.851430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjs99\" (UniqueName: \"kubernetes.io/projected/4c70a145-c987-41fb-8dbf-41d766d1f61d-kube-api-access-hjs99\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.851834 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.851800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c70a145-c987-41fb-8dbf-41d766d1f61d-tmp\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.854729 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.854704    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/4c70a145-c987-41fb-8dbf-41d766d1f61d-klusterlet-config\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.861732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.861696    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjs99\" (UniqueName: \"kubernetes.io/projected/4c70a145-c987-41fb-8dbf-41d766d1f61d-kube-api-access-hjs99\") pod \"klusterlet-addon-workmgr-d4d55c48c-d87bf\" (UID: \"4c70a145-c987-41fb-8dbf-41d766d1f61d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:22.885586 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.885548    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"
Sep 10 21:03:22.914899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:22.914860    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:23.168868 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:23.168837    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx"]
Sep 10 21:03:23.172211 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:03:23.172171    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod677e8f24_02f8_4c81_97ff_d9deeebb665c.slice/crio-366bfacd58491fb6fd2678438a27def0834833ef257d2bf9fc8fb184f1640d50 WatchSource:0}: Error finding container 366bfacd58491fb6fd2678438a27def0834833ef257d2bf9fc8fb184f1640d50: Status 404 returned error can't find the container with id 366bfacd58491fb6fd2678438a27def0834833ef257d2bf9fc8fb184f1640d50
Sep 10 21:03:23.201427 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:23.201396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"]
Sep 10 21:03:23.204492 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:03:23.204453    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c70a145_c987_41fb_8dbf_41d766d1f61d.slice/crio-c936476b39319f5d971fcf87e427af21b3133f430b2339ef01a96680b0936275 WatchSource:0}: Error finding container c936476b39319f5d971fcf87e427af21b3133f430b2339ef01a96680b0936275: Status 404 returned error can't find the container with id c936476b39319f5d971fcf87e427af21b3133f430b2339ef01a96680b0936275
Sep 10 21:03:23.683183 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:23.683136    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf" event={"ID":"4c70a145-c987-41fb-8dbf-41d766d1f61d","Type":"ContainerStarted","Data":"c936476b39319f5d971fcf87e427af21b3133f430b2339ef01a96680b0936275"}
Sep 10 21:03:23.684541 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:23.684488    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx" event={"ID":"677e8f24-02f8-4c81-97ff-d9deeebb665c","Type":"ContainerStarted","Data":"366bfacd58491fb6fd2678438a27def0834833ef257d2bf9fc8fb184f1640d50"}
Sep 10 21:03:30.715688 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:30.715653    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf" event={"ID":"4c70a145-c987-41fb-8dbf-41d766d1f61d","Type":"ContainerStarted","Data":"0d45c22eac2fe04beb258014b31c1afc51165a2668e17305d2c7d66714615569"}
Sep 10 21:03:30.716156 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:30.715839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:30.718005 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:30.717985    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf"
Sep 10 21:03:30.718156 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:30.718136    2580 generic.go:356] "Generic (PLEG): container finished" podID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerID="f137e202cf98582edb4ac219bdf44d28db18b752642763b6ce4e7ab516528d85" exitCode=0
Sep 10 21:03:30.718248 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:30.718224    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg" event={"ID":"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97","Type":"ContainerDied","Data":"f137e202cf98582edb4ac219bdf44d28db18b752642763b6ce4e7ab516528d85"}
Sep 10 21:03:30.719670 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:30.719645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx" event={"ID":"677e8f24-02f8-4c81-97ff-d9deeebb665c","Type":"ContainerStarted","Data":"44fbc1b19bf3f5f3573bbbad678d5e5c5690c53d7518e80d10be3230e63fae49"}
Sep 10 21:03:30.732395 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:30.732349    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-d4d55c48c-d87bf" podStartSLOduration=2.247214212 podStartE2EDuration="8.732330656s" totalImagesPullingTime="6.485116444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:30.732145963 +0000 UTC m=+297.734392302" watchObservedRunningTime="2026-09-10 21:03:30.732330656 +0000 UTC m=+297.734577004"
Sep 10 21:03:30.782572 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:30.782523    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6c9d79d558-87fzx" podStartSLOduration=2.280264648 podStartE2EDuration="8.782506678s" totalImagesPullingTime="6.50224203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:30.781584915 +0000 UTC m=+297.783831275" watchObservedRunningTime="2026-09-10 21:03:30.782506678 +0000 UTC m=+297.784753026"
Sep 10 21:03:31.897248 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:31.897220    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:32.030785 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.030712    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks65m\" (UniqueName: \"kubernetes.io/projected/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-kube-api-access-ks65m\") pod \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") "
Sep 10 21:03:32.030948 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.030789    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-bundle\") pod \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") "
Sep 10 21:03:32.030948 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.030818    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-util\") pod \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\" (UID: \"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97\") "
Sep 10 21:03:32.031440 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.031418    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-bundle" (OuterVolumeSpecName: "bundle") pod "fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" (UID: "fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:03:32.033083 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.033062    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-kube-api-access-ks65m" (OuterVolumeSpecName: "kube-api-access-ks65m") pod "fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" (UID: "fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97"). InnerVolumeSpecName "kube-api-access-ks65m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:03:32.035580 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.035559    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-util" (OuterVolumeSpecName: "util") pod "fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" (UID: "fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:03:32.131247 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.131220    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:03:32.131247 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.131242    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks65m\" (UniqueName: \"kubernetes.io/projected/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-kube-api-access-ks65m\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:03:32.131247 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.131252    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:03:32.728491 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.728459    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg"
Sep 10 21:03:32.728658 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.728495    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540kxfpg" event={"ID":"fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97","Type":"ContainerDied","Data":"ccc1966a2cd49640751b80f96a586cb55d94d11e4a17fcbe52cf814891cfed88"}
Sep 10 21:03:32.728658 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:32.728541    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccc1966a2cd49640751b80f96a586cb55d94d11e4a17fcbe52cf814891cfed88"
Sep 10 21:03:33.415831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:33.415811    2580 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 21:03:36.544551 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.544518    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"]
Sep 10 21:03:36.546324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.544764    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerName="extract"
Sep 10 21:03:36.546324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.544776    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerName="extract"
Sep 10 21:03:36.546324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.544791    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerName="util"
Sep 10 21:03:36.546324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.544796    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerName="util"
Sep 10 21:03:36.546324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.544807    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerName="pull"
Sep 10 21:03:36.546324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.544813    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerName="pull"
Sep 10 21:03:36.546324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.544867    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa24fbe8-195e-4a93-b6ac-2cc80e4b4a97" containerName="extract"
Sep 10 21:03:36.546743 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.546728    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"
Sep 10 21:03:36.548837 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.548813    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Sep 10 21:03:36.549008 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.548990    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-rwsfz\""
Sep 10 21:03:36.549083 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.549061    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:03:36.561781 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.561759    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"]
Sep 10 21:03:36.662901 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.662871    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90d38641-f22e-45f5-9084-841bca368b58-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dpfg5\" (UID: \"90d38641-f22e-45f5-9084-841bca368b58\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"
Sep 10 21:03:36.663035 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.662915    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td9pl\" (UniqueName: \"kubernetes.io/projected/90d38641-f22e-45f5-9084-841bca368b58-kube-api-access-td9pl\") pod \"cert-manager-operator-controller-manager-55c444bf86-dpfg5\" (UID: \"90d38641-f22e-45f5-9084-841bca368b58\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"
Sep 10 21:03:36.763902 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.763878    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90d38641-f22e-45f5-9084-841bca368b58-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dpfg5\" (UID: \"90d38641-f22e-45f5-9084-841bca368b58\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"
Sep 10 21:03:36.764025 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.763925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td9pl\" (UniqueName: \"kubernetes.io/projected/90d38641-f22e-45f5-9084-841bca368b58-kube-api-access-td9pl\") pod \"cert-manager-operator-controller-manager-55c444bf86-dpfg5\" (UID: \"90d38641-f22e-45f5-9084-841bca368b58\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"
Sep 10 21:03:36.764254 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.764237    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/90d38641-f22e-45f5-9084-841bca368b58-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dpfg5\" (UID: \"90d38641-f22e-45f5-9084-841bca368b58\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"
Sep 10 21:03:36.772322 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.772296    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-td9pl\" (UniqueName: \"kubernetes.io/projected/90d38641-f22e-45f5-9084-841bca368b58-kube-api-access-td9pl\") pod \"cert-manager-operator-controller-manager-55c444bf86-dpfg5\" (UID: \"90d38641-f22e-45f5-9084-841bca368b58\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"
Sep 10 21:03:36.855460 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:36.855391    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"
Sep 10 21:03:37.140030 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:37.139969    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5"]
Sep 10 21:03:37.143398 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:03:37.143369    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d38641_f22e_45f5_9084_841bca368b58.slice/crio-bd6a0b0662f5242377e27e5ca5f03af829fe5297f92575c4abc8d60d4a151100 WatchSource:0}: Error finding container bd6a0b0662f5242377e27e5ca5f03af829fe5297f92575c4abc8d60d4a151100: Status 404 returned error can't find the container with id bd6a0b0662f5242377e27e5ca5f03af829fe5297f92575c4abc8d60d4a151100
Sep 10 21:03:37.146172 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:37.146156    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:03:37.747670 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:37.747637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5" event={"ID":"90d38641-f22e-45f5-9084-841bca368b58","Type":"ContainerStarted","Data":"bd6a0b0662f5242377e27e5ca5f03af829fe5297f92575c4abc8d60d4a151100"}
Sep 10 21:03:39.757837 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:39.757802    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5" event={"ID":"90d38641-f22e-45f5-9084-841bca368b58","Type":"ContainerStarted","Data":"1a265b28ee59f8d9eb4c5e9b3e091b6be7fd14ffbd5ab8fb6c5fbf29bd109726"}
Sep 10 21:03:39.777286 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:39.777243    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dpfg5" podStartSLOduration=2.075978503 podStartE2EDuration="3.777229159s" totalImagesPullingTime="1.701250656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:39.776556924 +0000 UTC m=+306.778803268" watchObservedRunningTime="2026-09-10 21:03:39.777229159 +0000 UTC m=+306.779475505"
Sep 10 21:03:41.079318 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.079283    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"]
Sep 10 21:03:41.082014 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.081996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.084062 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.084043    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 21:03:41.084187 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.084067    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jvspl\""
Sep 10 21:03:41.084656 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.084644    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 21:03:41.092189 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.092159    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"]
Sep 10 21:03:41.196321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.196289    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.196321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.196323    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.196521 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.196361    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n9wv\" (UniqueName: \"kubernetes.io/projected/f4b8433e-11cd-4437-93bf-414b64498953-kube-api-access-9n9wv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.297604 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.297574    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.297715 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.297608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.297715 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.297639    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n9wv\" (UniqueName: \"kubernetes.io/projected/f4b8433e-11cd-4437-93bf-414b64498953-kube-api-access-9n9wv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.297987 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.297968    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.298022 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.297976    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.306673 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.306635    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n9wv\" (UniqueName: \"kubernetes.io/projected/f4b8433e-11cd-4437-93bf-414b64498953-kube-api-access-9n9wv\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.397215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.397196    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:41.667629 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.667601    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"]
Sep 10 21:03:41.669886 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:03:41.669859    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b8433e_11cd_4437_93bf_414b64498953.slice/crio-96716d8fabd3bc703ad52350dda2a2a69cd8deb286ab2d0520ba2ab0dfddfe3d WatchSource:0}: Error finding container 96716d8fabd3bc703ad52350dda2a2a69cd8deb286ab2d0520ba2ab0dfddfe3d: Status 404 returned error can't find the container with id 96716d8fabd3bc703ad52350dda2a2a69cd8deb286ab2d0520ba2ab0dfddfe3d
Sep 10 21:03:41.767052 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.767016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw" event={"ID":"f4b8433e-11cd-4437-93bf-414b64498953","Type":"ContainerStarted","Data":"7c209961932a8dc426020a345815fd5d05cb4cd4281e5e0d8e4a9d9058ce94b9"}
Sep 10 21:03:41.767197 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:41.767072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw" event={"ID":"f4b8433e-11cd-4437-93bf-414b64498953","Type":"ContainerStarted","Data":"96716d8fabd3bc703ad52350dda2a2a69cd8deb286ab2d0520ba2ab0dfddfe3d"}
Sep 10 21:03:42.772789 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:42.772740    2580 generic.go:356] "Generic (PLEG): container finished" podID="f4b8433e-11cd-4437-93bf-414b64498953" containerID="7c209961932a8dc426020a345815fd5d05cb4cd4281e5e0d8e4a9d9058ce94b9" exitCode=0
Sep 10 21:03:42.773238 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:42.772812    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw" event={"ID":"f4b8433e-11cd-4437-93bf-414b64498953","Type":"ContainerDied","Data":"7c209961932a8dc426020a345815fd5d05cb4cd4281e5e0d8e4a9d9058ce94b9"}
Sep 10 21:03:44.783081 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:44.783041    2580 generic.go:356] "Generic (PLEG): container finished" podID="f4b8433e-11cd-4437-93bf-414b64498953" containerID="5c5be62eb93e19baee2eaec1c718abc1ae62390c86d4a79733c8cfee6098a313" exitCode=0
Sep 10 21:03:44.783489 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:44.783137    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw" event={"ID":"f4b8433e-11cd-4437-93bf-414b64498953","Type":"ContainerDied","Data":"5c5be62eb93e19baee2eaec1c718abc1ae62390c86d4a79733c8cfee6098a313"}
Sep 10 21:03:45.492222 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.492186    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"]
Sep 10 21:03:45.494389 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.494371    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"
Sep 10 21:03:45.496395 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.496373    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 21:03:45.497084 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.497067    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-t7glb\""
Sep 10 21:03:45.497203 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.497107    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 21:03:45.504787 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.504765    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"]
Sep 10 21:03:45.631742 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.631708    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/831a0688-8c6b-47b6-9ccf-b8db4dafea0e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qbw9n\" (UID: \"831a0688-8c6b-47b6-9ccf-b8db4dafea0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"
Sep 10 21:03:45.631934 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.631755    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wvxz\" (UniqueName: \"kubernetes.io/projected/831a0688-8c6b-47b6-9ccf-b8db4dafea0e-kube-api-access-6wvxz\") pod \"cert-manager-cainjector-559cdd8c87-qbw9n\" (UID: \"831a0688-8c6b-47b6-9ccf-b8db4dafea0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"
Sep 10 21:03:45.732675 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.732647    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/831a0688-8c6b-47b6-9ccf-b8db4dafea0e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qbw9n\" (UID: \"831a0688-8c6b-47b6-9ccf-b8db4dafea0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"
Sep 10 21:03:45.732836 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.732683    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wvxz\" (UniqueName: \"kubernetes.io/projected/831a0688-8c6b-47b6-9ccf-b8db4dafea0e-kube-api-access-6wvxz\") pod \"cert-manager-cainjector-559cdd8c87-qbw9n\" (UID: \"831a0688-8c6b-47b6-9ccf-b8db4dafea0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"
Sep 10 21:03:45.740714 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.740694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/831a0688-8c6b-47b6-9ccf-b8db4dafea0e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-qbw9n\" (UID: \"831a0688-8c6b-47b6-9ccf-b8db4dafea0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"
Sep 10 21:03:45.740794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.740757    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wvxz\" (UniqueName: \"kubernetes.io/projected/831a0688-8c6b-47b6-9ccf-b8db4dafea0e-kube-api-access-6wvxz\") pod \"cert-manager-cainjector-559cdd8c87-qbw9n\" (UID: \"831a0688-8c6b-47b6-9ccf-b8db4dafea0e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"
Sep 10 21:03:45.788921 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.788860    2580 generic.go:356] "Generic (PLEG): container finished" podID="f4b8433e-11cd-4437-93bf-414b64498953" containerID="861c445b386231a8fa9ef2e84aee0ee23e8c7db3b39e0139ee086187f28230be" exitCode=0
Sep 10 21:03:45.789237 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.788947    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw" event={"ID":"f4b8433e-11cd-4437-93bf-414b64498953","Type":"ContainerDied","Data":"861c445b386231a8fa9ef2e84aee0ee23e8c7db3b39e0139ee086187f28230be"}
Sep 10 21:03:45.803235 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:45.803218    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"
Sep 10 21:03:46.072272 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:46.072050    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n"]
Sep 10 21:03:46.074419 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:03:46.074393    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod831a0688_8c6b_47b6_9ccf_b8db4dafea0e.slice/crio-b9bfe0663f36fd9a8c7167c875a3732cd38ad2aeb35a261cd4267bf6de110140 WatchSource:0}: Error finding container b9bfe0663f36fd9a8c7167c875a3732cd38ad2aeb35a261cd4267bf6de110140: Status 404 returned error can't find the container with id b9bfe0663f36fd9a8c7167c875a3732cd38ad2aeb35a261cd4267bf6de110140
Sep 10 21:03:46.799474 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:46.799440    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n" event={"ID":"831a0688-8c6b-47b6-9ccf-b8db4dafea0e","Type":"ContainerStarted","Data":"b9bfe0663f36fd9a8c7167c875a3732cd38ad2aeb35a261cd4267bf6de110140"}
Sep 10 21:03:46.923764 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:46.923728    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:47.043652 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.043616    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-bundle\") pod \"f4b8433e-11cd-4437-93bf-414b64498953\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") "
Sep 10 21:03:47.043828 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.043658    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9n9wv\" (UniqueName: \"kubernetes.io/projected/f4b8433e-11cd-4437-93bf-414b64498953-kube-api-access-9n9wv\") pod \"f4b8433e-11cd-4437-93bf-414b64498953\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") "
Sep 10 21:03:47.043828 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.043711    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-util\") pod \"f4b8433e-11cd-4437-93bf-414b64498953\" (UID: \"f4b8433e-11cd-4437-93bf-414b64498953\") "
Sep 10 21:03:47.044163 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.044092    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-bundle" (OuterVolumeSpecName: "bundle") pod "f4b8433e-11cd-4437-93bf-414b64498953" (UID: "f4b8433e-11cd-4437-93bf-414b64498953"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:03:47.046583 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.046554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b8433e-11cd-4437-93bf-414b64498953-kube-api-access-9n9wv" (OuterVolumeSpecName: "kube-api-access-9n9wv") pod "f4b8433e-11cd-4437-93bf-414b64498953" (UID: "f4b8433e-11cd-4437-93bf-414b64498953"). InnerVolumeSpecName "kube-api-access-9n9wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:03:47.051563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.051455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-util" (OuterVolumeSpecName: "util") pod "f4b8433e-11cd-4437-93bf-414b64498953" (UID: "f4b8433e-11cd-4437-93bf-414b64498953"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:03:47.145381 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.145342    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n9wv\" (UniqueName: \"kubernetes.io/projected/f4b8433e-11cd-4437-93bf-414b64498953-kube-api-access-9n9wv\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:03:47.145381 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.145380    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:03:47.145613 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.145395    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4b8433e-11cd-4437-93bf-414b64498953-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:03:47.806273 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.806197    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw" event={"ID":"f4b8433e-11cd-4437-93bf-414b64498953","Type":"ContainerDied","Data":"96716d8fabd3bc703ad52350dda2a2a69cd8deb286ab2d0520ba2ab0dfddfe3d"}
Sep 10 21:03:47.806273 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.806243    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96716d8fabd3bc703ad52350dda2a2a69cd8deb286ab2d0520ba2ab0dfddfe3d"
Sep 10 21:03:47.806721 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:47.806269    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fjj4lw"
Sep 10 21:03:49.816066 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:49.816033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n" event={"ID":"831a0688-8c6b-47b6-9ccf-b8db4dafea0e","Type":"ContainerStarted","Data":"b58296311d0027242d8825252e36ffd1405a64a933fa81c55476603806eb13cc"}
Sep 10 21:03:49.833823 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:49.833780    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-qbw9n" podStartSLOduration=2.093894688 podStartE2EDuration="4.833765304s" totalImagesPullingTime="2.739870616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:49.832638132 +0000 UTC m=+316.834884479" watchObservedRunningTime="2026-09-10 21:03:49.833765304 +0000 UTC m=+316.836011651"
Sep 10 21:03:52.063178 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.063142    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6n49m"]
Sep 10 21:03:52.063547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.063398    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b8433e-11cd-4437-93bf-414b64498953" containerName="util"
Sep 10 21:03:52.063547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.063409    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b8433e-11cd-4437-93bf-414b64498953" containerName="util"
Sep 10 21:03:52.063547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.063424    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b8433e-11cd-4437-93bf-414b64498953" containerName="pull"
Sep 10 21:03:52.063547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.063430    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b8433e-11cd-4437-93bf-414b64498953" containerName="pull"
Sep 10 21:03:52.063547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.063440    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b8433e-11cd-4437-93bf-414b64498953" containerName="extract"
Sep 10 21:03:52.063547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.063445    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b8433e-11cd-4437-93bf-414b64498953" containerName="extract"
Sep 10 21:03:52.063547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.063499    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b8433e-11cd-4437-93bf-414b64498953" containerName="extract"
Sep 10 21:03:52.065387 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.065372    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6n49m"
Sep 10 21:03:52.067414 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.067394    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-vpqht\""
Sep 10 21:03:52.076027 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.076006    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6n49m"]
Sep 10 21:03:52.184156 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.184104    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqz5x\" (UniqueName: \"kubernetes.io/projected/ae958980-7332-4152-837b-b09bc433990c-kube-api-access-nqz5x\") pod \"cert-manager-55b4d954d5-6n49m\" (UID: \"ae958980-7332-4152-837b-b09bc433990c\") " pod="cert-manager/cert-manager-55b4d954d5-6n49m"
Sep 10 21:03:52.184312 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.184205    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae958980-7332-4152-837b-b09bc433990c-bound-sa-token\") pod \"cert-manager-55b4d954d5-6n49m\" (UID: \"ae958980-7332-4152-837b-b09bc433990c\") " pod="cert-manager/cert-manager-55b4d954d5-6n49m"
Sep 10 21:03:52.285411 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.285379    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae958980-7332-4152-837b-b09bc433990c-bound-sa-token\") pod \"cert-manager-55b4d954d5-6n49m\" (UID: \"ae958980-7332-4152-837b-b09bc433990c\") " pod="cert-manager/cert-manager-55b4d954d5-6n49m"
Sep 10 21:03:52.285576 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.285452    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqz5x\" (UniqueName: \"kubernetes.io/projected/ae958980-7332-4152-837b-b09bc433990c-kube-api-access-nqz5x\") pod \"cert-manager-55b4d954d5-6n49m\" (UID: \"ae958980-7332-4152-837b-b09bc433990c\") " pod="cert-manager/cert-manager-55b4d954d5-6n49m"
Sep 10 21:03:52.293113 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.293087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ae958980-7332-4152-837b-b09bc433990c-bound-sa-token\") pod \"cert-manager-55b4d954d5-6n49m\" (UID: \"ae958980-7332-4152-837b-b09bc433990c\") " pod="cert-manager/cert-manager-55b4d954d5-6n49m"
Sep 10 21:03:52.293234 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.293094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqz5x\" (UniqueName: \"kubernetes.io/projected/ae958980-7332-4152-837b-b09bc433990c-kube-api-access-nqz5x\") pod \"cert-manager-55b4d954d5-6n49m\" (UID: \"ae958980-7332-4152-837b-b09bc433990c\") " pod="cert-manager/cert-manager-55b4d954d5-6n49m"
Sep 10 21:03:52.373782 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.373759    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-6n49m"
Sep 10 21:03:52.644922 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.644901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-6n49m"]
Sep 10 21:03:52.647242 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:03:52.647214    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae958980_7332_4152_837b_b09bc433990c.slice/crio-5e93ea3766104a57a470d7d9c973bb33c35b4919eeb75c2e1ab15b1e543b7861 WatchSource:0}: Error finding container 5e93ea3766104a57a470d7d9c973bb33c35b4919eeb75c2e1ab15b1e543b7861: Status 404 returned error can't find the container with id 5e93ea3766104a57a470d7d9c973bb33c35b4919eeb75c2e1ab15b1e543b7861
Sep 10 21:03:52.829212 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.829161    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6n49m" event={"ID":"ae958980-7332-4152-837b-b09bc433990c","Type":"ContainerStarted","Data":"094ab9807b8c8365fcdba581e995080b5513abd8c1a3bceaf9d81d2e08008938"}
Sep 10 21:03:52.829401 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.829222    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-6n49m" event={"ID":"ae958980-7332-4152-837b-b09bc433990c","Type":"ContainerStarted","Data":"5e93ea3766104a57a470d7d9c973bb33c35b4919eeb75c2e1ab15b1e543b7861"}
Sep 10 21:03:52.846019 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:03:52.845975    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-6n49m" podStartSLOduration=0.845960432 podStartE2EDuration="845.960432ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:03:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:03:52.844732424 +0000 UTC m=+319.846978772" watchObservedRunningTime="2026-09-10 21:03:52.845960432 +0000 UTC m=+319.848206779"
Sep 10 21:04:01.418626 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.418590    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"]
Sep 10 21:04:01.470363 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.470333    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"]
Sep 10 21:04:01.470537 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.470458    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.473667 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.473643    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 21:04:01.473788 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.473672    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jvspl\""
Sep 10 21:04:01.473913 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.473901    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 21:04:01.546827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.546778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.546827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.546808    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.547016 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.546891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f26gq\" (UniqueName: \"kubernetes.io/projected/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-kube-api-access-f26gq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.647625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.647587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f26gq\" (UniqueName: \"kubernetes.io/projected/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-kube-api-access-f26gq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.647794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.647648    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.647794 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.647673    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.648050 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.648032    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.648087 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.648057    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.656042 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.656022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f26gq\" (UniqueName: \"kubernetes.io/projected/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-kube-api-access-f26gq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:01.779827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:01.779742    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:02.083459 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:02.083438    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"]
Sep 10 21:04:02.085905 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:04:02.085875    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5dbc7b_0467_4aec_bfa3_3a7ae161d656.slice/crio-dfe470a2ed7e2e4989bc0e877d9eec7272f5cdda0b85f0cb2925e621f56c688e WatchSource:0}: Error finding container dfe470a2ed7e2e4989bc0e877d9eec7272f5cdda0b85f0cb2925e621f56c688e: Status 404 returned error can't find the container with id dfe470a2ed7e2e4989bc0e877d9eec7272f5cdda0b85f0cb2925e621f56c688e
Sep 10 21:04:02.873216 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:02.873181    2580 generic.go:356] "Generic (PLEG): container finished" podID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerID="f7a2b6d2284c81a1f9ada996a8c50d2edd242d5b30bb638885e110344c6d184d" exitCode=0
Sep 10 21:04:02.873593 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:02.873245    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk" event={"ID":"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656","Type":"ContainerDied","Data":"f7a2b6d2284c81a1f9ada996a8c50d2edd242d5b30bb638885e110344c6d184d"}
Sep 10 21:04:02.873593 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:02.873284    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk" event={"ID":"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656","Type":"ContainerStarted","Data":"dfe470a2ed7e2e4989bc0e877d9eec7272f5cdda0b85f0cb2925e621f56c688e"}
Sep 10 21:04:04.883162 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:04.883110    2580 generic.go:356] "Generic (PLEG): container finished" podID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerID="1179bee3b7572494b21c02b589b177690e748e5f3a609b9fc7e1c385f90be6fc" exitCode=0
Sep 10 21:04:04.883162 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:04.883150    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk" event={"ID":"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656","Type":"ContainerDied","Data":"1179bee3b7572494b21c02b589b177690e748e5f3a609b9fc7e1c385f90be6fc"}
Sep 10 21:04:05.889681 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:05.889649    2580 generic.go:356] "Generic (PLEG): container finished" podID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerID="a6b5a6582baab42197dbcf5c7bc28e41fa842dc35dbb595fbdfaa30efef28db4" exitCode=0
Sep 10 21:04:05.890031 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:05.889720    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk" event={"ID":"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656","Type":"ContainerDied","Data":"a6b5a6582baab42197dbcf5c7bc28e41fa842dc35dbb595fbdfaa30efef28db4"}
Sep 10 21:04:07.014535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.014512    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:07.084181 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.084150    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f26gq\" (UniqueName: \"kubernetes.io/projected/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-kube-api-access-f26gq\") pod \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") "
Sep 10 21:04:07.084354 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.084196    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-util\") pod \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") "
Sep 10 21:04:07.084354 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.084223    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-bundle\") pod \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\" (UID: \"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656\") "
Sep 10 21:04:07.084990 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.084957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-bundle" (OuterVolumeSpecName: "bundle") pod "3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" (UID: "3a5dbc7b-0467-4aec-bfa3-3a7ae161d656"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:07.086421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.086397    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-kube-api-access-f26gq" (OuterVolumeSpecName: "kube-api-access-f26gq") pod "3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" (UID: "3a5dbc7b-0467-4aec-bfa3-3a7ae161d656"). InnerVolumeSpecName "kube-api-access-f26gq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:07.089686 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.089649    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-util" (OuterVolumeSpecName: "util") pod "3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" (UID: "3a5dbc7b-0467-4aec-bfa3-3a7ae161d656"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:07.185503 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.185435    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f26gq\" (UniqueName: \"kubernetes.io/projected/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-kube-api-access-f26gq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:04:07.185503 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.185460    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:04:07.185503 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.185469    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3a5dbc7b-0467-4aec-bfa3-3a7ae161d656-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:04:07.899922 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.899889    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk" event={"ID":"3a5dbc7b-0467-4aec-bfa3-3a7ae161d656","Type":"ContainerDied","Data":"dfe470a2ed7e2e4989bc0e877d9eec7272f5cdda0b85f0cb2925e621f56c688e"}
Sep 10 21:04:07.900095 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.899926    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afktnlk"
Sep 10 21:04:07.900095 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:07.899940    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfe470a2ed7e2e4989bc0e877d9eec7272f5cdda0b85f0cb2925e621f56c688e"
Sep 10 21:04:17.073996 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.073961    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"]
Sep 10 21:04:17.074380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.074249    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerName="extract"
Sep 10 21:04:17.074380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.074260    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerName="extract"
Sep 10 21:04:17.074380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.074272    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerName="pull"
Sep 10 21:04:17.074380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.074278    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerName="pull"
Sep 10 21:04:17.074380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.074292    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerName="util"
Sep 10 21:04:17.074380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.074297    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerName="util"
Sep 10 21:04:17.074380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.074341    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a5dbc7b-0467-4aec-bfa3-3a7ae161d656" containerName="extract"
Sep 10 21:04:17.077132 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.077105    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.079618 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.079596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 21:04:17.079752 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.079652    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jvspl\""
Sep 10 21:04:17.080149 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.080136    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 21:04:17.088561 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.088539    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"]
Sep 10 21:04:17.157860 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.157832    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.158006 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.157863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnc5q\" (UniqueName: \"kubernetes.io/projected/5e8ad497-7f01-4221-b1aa-723226406c38-kube-api-access-dnc5q\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.158006 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.157888    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.259245 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.259204    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc5q\" (UniqueName: \"kubernetes.io/projected/5e8ad497-7f01-4221-b1aa-723226406c38-kube-api-access-dnc5q\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.259429 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.259256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.259429 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.259334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.259657 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.259636    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.259737 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.259694    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.267849 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.267829    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnc5q\" (UniqueName: \"kubernetes.io/projected/5e8ad497-7f01-4221-b1aa-723226406c38-kube-api-access-dnc5q\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.385656 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.385628    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:17.662310 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.662231    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"]
Sep 10 21:04:17.665473 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:04:17.665436    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8ad497_7f01_4221_b1aa_723226406c38.slice/crio-18f799d0878a6c192230e12f5190ae55c6049bbc1ef075af7908fe1a2bd6c065 WatchSource:0}: Error finding container 18f799d0878a6c192230e12f5190ae55c6049bbc1ef075af7908fe1a2bd6c065: Status 404 returned error can't find the container with id 18f799d0878a6c192230e12f5190ae55c6049bbc1ef075af7908fe1a2bd6c065
Sep 10 21:04:17.944049 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.943974    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e8ad497-7f01-4221-b1aa-723226406c38" containerID="e0ce08779d7a1b70a0eb6a9b3ffb4dd8e6f40122d8e4c4ecc167111220a32a9b" exitCode=0
Sep 10 21:04:17.944049 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.944033    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx" event={"ID":"5e8ad497-7f01-4221-b1aa-723226406c38","Type":"ContainerDied","Data":"e0ce08779d7a1b70a0eb6a9b3ffb4dd8e6f40122d8e4c4ecc167111220a32a9b"}
Sep 10 21:04:17.944255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:17.944068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx" event={"ID":"5e8ad497-7f01-4221-b1aa-723226406c38","Type":"ContainerStarted","Data":"18f799d0878a6c192230e12f5190ae55c6049bbc1ef075af7908fe1a2bd6c065"}
Sep 10 21:04:18.735551 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.735515    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"]
Sep 10 21:04:18.738660 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.738644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.741751 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.741732    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 21:04:18.741912 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.741896    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 21:04:18.742011 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.741995    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 21:04:18.742235 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.742220    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 21:04:18.742635 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.742603    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cv4zb\""
Sep 10 21:04:18.750943 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.750918    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"]
Sep 10 21:04:18.772220 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.772197    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-tmp\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.772323 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.772282    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.772323 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.772303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-webhook-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.772449 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.772373    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-manifests\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.772449 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.772406    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2pk4\" (UniqueName: \"kubernetes.io/projected/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-kube-api-access-v2pk4\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.872895 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.872826    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-tmp\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.872895 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.872875    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.872895 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.872893    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-webhook-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.873162 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.872918    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-manifests\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.873162 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.872945    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2pk4\" (UniqueName: \"kubernetes.io/projected/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-kube-api-access-v2pk4\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.873334 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.873316    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-manifests\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.875344 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.875318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-tmp\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.875462 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.875371    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-webhook-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.875518 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.875468    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-apiservice-cert\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.885184 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.885164    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2pk4\" (UniqueName: \"kubernetes.io/projected/e2f792bf-bba1-4120-b9cb-7d4d9f64ab37-kube-api-access-v2pk4\") pod \"opendatahub-operator-controller-manager-d67fdbbcb-j64nn\" (UID: \"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37\") " pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:18.950557 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.950527    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e8ad497-7f01-4221-b1aa-723226406c38" containerID="ee2cea0c8b174ed65177b70730ed4b1abaedae36a768634c045651451dfed995" exitCode=0
Sep 10 21:04:18.950731 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:18.950582    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx" event={"ID":"5e8ad497-7f01-4221-b1aa-723226406c38","Type":"ContainerDied","Data":"ee2cea0c8b174ed65177b70730ed4b1abaedae36a768634c045651451dfed995"}
Sep 10 21:04:19.047955 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:19.047925    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:19.338271 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:19.338248    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"]
Sep 10 21:04:19.340477 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:04:19.340449    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f792bf_bba1_4120_b9cb_7d4d9f64ab37.slice/crio-0d54dd5ffc47f2c577c00b2e0db5298213f3294c45f7671c22a4a065fc2ecb24 WatchSource:0}: Error finding container 0d54dd5ffc47f2c577c00b2e0db5298213f3294c45f7671c22a4a065fc2ecb24: Status 404 returned error can't find the container with id 0d54dd5ffc47f2c577c00b2e0db5298213f3294c45f7671c22a4a065fc2ecb24
Sep 10 21:04:19.956503 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:19.956464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn" event={"ID":"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37","Type":"ContainerStarted","Data":"0d54dd5ffc47f2c577c00b2e0db5298213f3294c45f7671c22a4a065fc2ecb24"}
Sep 10 21:04:19.958533 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:19.958510    2580 generic.go:356] "Generic (PLEG): container finished" podID="5e8ad497-7f01-4221-b1aa-723226406c38" containerID="656185cced575e204f9f423893f6390682c39f610e96c5d6e2c6818ac27559e7" exitCode=0
Sep 10 21:04:19.958660 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:19.958597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx" event={"ID":"5e8ad497-7f01-4221-b1aa-723226406c38","Type":"ContainerDied","Data":"656185cced575e204f9f423893f6390682c39f610e96c5d6e2c6818ac27559e7"}
Sep 10 21:04:22.400364 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.400340    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:22.503337 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.503315    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnc5q\" (UniqueName: \"kubernetes.io/projected/5e8ad497-7f01-4221-b1aa-723226406c38-kube-api-access-dnc5q\") pod \"5e8ad497-7f01-4221-b1aa-723226406c38\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") "
Sep 10 21:04:22.503431 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.503384    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-bundle\") pod \"5e8ad497-7f01-4221-b1aa-723226406c38\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") "
Sep 10 21:04:22.503553 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.503533    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-util\") pod \"5e8ad497-7f01-4221-b1aa-723226406c38\" (UID: \"5e8ad497-7f01-4221-b1aa-723226406c38\") "
Sep 10 21:04:22.504074 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.504052    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-bundle" (OuterVolumeSpecName: "bundle") pod "5e8ad497-7f01-4221-b1aa-723226406c38" (UID: "5e8ad497-7f01-4221-b1aa-723226406c38"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:22.505339 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.505316    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8ad497-7f01-4221-b1aa-723226406c38-kube-api-access-dnc5q" (OuterVolumeSpecName: "kube-api-access-dnc5q") pod "5e8ad497-7f01-4221-b1aa-723226406c38" (UID: "5e8ad497-7f01-4221-b1aa-723226406c38"). InnerVolumeSpecName "kube-api-access-dnc5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:04:22.511365 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.511329    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-util" (OuterVolumeSpecName: "util") pod "5e8ad497-7f01-4221-b1aa-723226406c38" (UID: "5e8ad497-7f01-4221-b1aa-723226406c38"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:04:22.604823 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.604777    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:04:22.604823 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.604816    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e8ad497-7f01-4221-b1aa-723226406c38-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:04:22.604823 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.604825    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnc5q\" (UniqueName: \"kubernetes.io/projected/5e8ad497-7f01-4221-b1aa-723226406c38-kube-api-access-dnc5q\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:04:22.973370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.973332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn" event={"ID":"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37","Type":"ContainerStarted","Data":"91fa4aa4d35a4b52e3581a472266d53efc0190ba136dba3e13062ef9131b794b"}
Sep 10 21:04:22.975202 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.975179    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx"
Sep 10 21:04:22.975347 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.975199    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b08xpmx" event={"ID":"5e8ad497-7f01-4221-b1aa-723226406c38","Type":"ContainerDied","Data":"18f799d0878a6c192230e12f5190ae55c6049bbc1ef075af7908fe1a2bd6c065"}
Sep 10 21:04:22.975347 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:22.975237    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f799d0878a6c192230e12f5190ae55c6049bbc1ef075af7908fe1a2bd6c065"
Sep 10 21:04:23.980989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:23.980953    2580 generic.go:356] "Generic (PLEG): container finished" podID="e2f792bf-bba1-4120-b9cb-7d4d9f64ab37" containerID="91fa4aa4d35a4b52e3581a472266d53efc0190ba136dba3e13062ef9131b794b" exitCode=0
Sep 10 21:04:23.981375 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:23.981024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn" event={"ID":"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37","Type":"ContainerDied","Data":"91fa4aa4d35a4b52e3581a472266d53efc0190ba136dba3e13062ef9131b794b"}
Sep 10 21:04:24.987140 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:24.987087    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn" event={"ID":"e2f792bf-bba1-4120-b9cb-7d4d9f64ab37","Type":"ContainerStarted","Data":"35f291b495ef21a3cdae879b080bac904e8b7f69f7ae9fd7efc9ec825c952614"}
Sep 10 21:04:24.987523 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:24.987235    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:25.009731 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:25.009681    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn" podStartSLOduration=2.027815575 podStartE2EDuration="7.009664914s" totalImagesPullingTime="3.896628834s" totalInitContainerRuntime="1.085220505s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:25.007546338 +0000 UTC m=+352.009792697" watchObservedRunningTime="2026-09-10 21:04:25.009664914 +0000 UTC m=+352.011911262"
Sep 10 21:04:35.992985 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:35.992956    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-d67fdbbcb-j64nn"
Sep 10 21:04:45.072171 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.072140    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"]
Sep 10 21:04:45.072571 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.072521    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8ad497-7f01-4221-b1aa-723226406c38" containerName="pull"
Sep 10 21:04:45.072571 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.072537    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8ad497-7f01-4221-b1aa-723226406c38" containerName="pull"
Sep 10 21:04:45.072571 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.072561    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8ad497-7f01-4221-b1aa-723226406c38" containerName="extract"
Sep 10 21:04:45.072571 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.072570    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8ad497-7f01-4221-b1aa-723226406c38" containerName="extract"
Sep 10 21:04:45.072694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.072590    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e8ad497-7f01-4221-b1aa-723226406c38" containerName="util"
Sep 10 21:04:45.072694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.072598    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e8ad497-7f01-4221-b1aa-723226406c38" containerName="util"
Sep 10 21:04:45.072694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.072666    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e8ad497-7f01-4221-b1aa-723226406c38" containerName="extract"
Sep 10 21:04:45.081940 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.081919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.084706 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.084687    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 21:04:45.084836 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.084819    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cfgz2\""
Sep 10 21:04:45.085018 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.085001    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 21:04:45.085018 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.085017    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:04:45.085177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.085038    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 21:04:45.090200 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.090178    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"]
Sep 10 21:04:45.170001 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.169978    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nl62j\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-kube-api-access-nl62j\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.170177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.170009    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.170177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.170031    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.170177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.170050    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.170177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.170067    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.170177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.170155    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.170408 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.170255    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.270799 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.270753    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nl62j\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-kube-api-access-nl62j\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.270799 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.270803    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.271038 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.270828    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.271038 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.270892    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.271038 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.270935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.271038 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.270964    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.271038 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.271019    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.271838 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.271811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.273353 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.273331    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.273461 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.273372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.273461 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.273422    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.273461 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.273454    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.279378 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.279357    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nl62j\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-kube-api-access-nl62j\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.279740 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.279723    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-mpvxw\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.392025 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.391993    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:45.665168 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:45.665143    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"]
Sep 10 21:04:45.667244 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:04:45.667223    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd015bf2b_47bd_4e1d_9e42_2ffe951da02e.slice/crio-27477c0a36d1735f45177ee9f10260025fe80375760e94e5e0ca73fb20cde324 WatchSource:0}: Error finding container 27477c0a36d1735f45177ee9f10260025fe80375760e94e5e0ca73fb20cde324: Status 404 returned error can't find the container with id 27477c0a36d1735f45177ee9f10260025fe80375760e94e5e0ca73fb20cde324
Sep 10 21:04:46.080572 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:46.080485    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw" event={"ID":"d015bf2b-47bd-4e1d-9e42-2ffe951da02e","Type":"ContainerStarted","Data":"27477c0a36d1735f45177ee9f10260025fe80375760e94e5e0ca73fb20cde324"}
Sep 10 21:04:48.241266 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:48.241220    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:04:48.241556 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:48.241297    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:04:49.097165 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:49.097115    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw" event={"ID":"d015bf2b-47bd-4e1d-9e42-2ffe951da02e","Type":"ContainerStarted","Data":"7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9"}
Sep 10 21:04:49.097339 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:49.097184    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:49.098894 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:49.098868    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-mpvxw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:04:49.099005 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:49.098919    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw" podUID="d015bf2b-47bd-4e1d-9e42-2ffe951da02e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:04:49.120855 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:49.120802    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw" podStartSLOduration=1.549522118 podStartE2EDuration="4.120786906s" totalImagesPullingTime="2.571264788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:04:49.118691555 +0000 UTC m=+376.120937901" watchObservedRunningTime="2026-09-10 21:04:49.120786906 +0000 UTC m=+376.123033253"
Sep 10 21:04:50.102956 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:50.102924    2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-mpvxw container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:04:50.103343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:50.102978    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw" podUID="d015bf2b-47bd-4e1d-9e42-2ffe951da02e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:04:53.102946 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:53.102912    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:04:56.965887 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.965853    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"]
Sep 10 21:04:56.972765 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.972745    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:56.975624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.975603    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 21:04:56.975717 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.975651    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 21:04:56.976377 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.976354    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 21:04:56.976458 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.976417    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 21:04:56.976511 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.976473    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-98fwg\""
Sep 10 21:04:56.976561 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.976551    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 21:04:56.981072 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:56.981052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"]
Sep 10 21:04:57.065336 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.065304    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e841aab1-748a-4125-a278-a9d9330fdd1c-metrics-cert\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.065493 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.065342    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgkjk\" (UniqueName: \"kubernetes.io/projected/e841aab1-748a-4125-a278-a9d9330fdd1c-kube-api-access-sgkjk\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.065493 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.065363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e841aab1-748a-4125-a278-a9d9330fdd1c-manager-config\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.065493 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.065390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e841aab1-748a-4125-a278-a9d9330fdd1c-cert\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.166728 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.166692    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgkjk\" (UniqueName: \"kubernetes.io/projected/e841aab1-748a-4125-a278-a9d9330fdd1c-kube-api-access-sgkjk\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.166728 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.166731    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e841aab1-748a-4125-a278-a9d9330fdd1c-manager-config\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.166974 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.166771    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e841aab1-748a-4125-a278-a9d9330fdd1c-cert\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.166974 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.166832    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e841aab1-748a-4125-a278-a9d9330fdd1c-metrics-cert\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.167525 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.167473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e841aab1-748a-4125-a278-a9d9330fdd1c-manager-config\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.169506 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.169486    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e841aab1-748a-4125-a278-a9d9330fdd1c-metrics-cert\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.169587 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.169533    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e841aab1-748a-4125-a278-a9d9330fdd1c-cert\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.179516 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.179494    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgkjk\" (UniqueName: \"kubernetes.io/projected/e841aab1-748a-4125-a278-a9d9330fdd1c-kube-api-access-sgkjk\") pod \"lws-controller-manager-548fd67646-q8cd6\" (UID: \"e841aab1-748a-4125-a278-a9d9330fdd1c\") " pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.282626 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.282538    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:04:57.563730 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:57.563636    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"]
Sep 10 21:04:57.566404 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:04:57.566376    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode841aab1_748a_4125_a278_a9d9330fdd1c.slice/crio-398b04568c281507e3ca8711b9bb210986e86002893dafe51ebe241fabaa939c WatchSource:0}: Error finding container 398b04568c281507e3ca8711b9bb210986e86002893dafe51ebe241fabaa939c: Status 404 returned error can't find the container with id 398b04568c281507e3ca8711b9bb210986e86002893dafe51ebe241fabaa939c
Sep 10 21:04:58.137406 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:04:58.137373    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6" event={"ID":"e841aab1-748a-4125-a278-a9d9330fdd1c","Type":"ContainerStarted","Data":"398b04568c281507e3ca8711b9bb210986e86002893dafe51ebe241fabaa939c"}
Sep 10 21:05:00.149210 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:00.149158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6" event={"ID":"e841aab1-748a-4125-a278-a9d9330fdd1c","Type":"ContainerStarted","Data":"3e64cf16840558385dbfab2a1f248773a541f09521c18553018667fa5ffd4cc9"}
Sep 10 21:05:00.149612 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:00.149362    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:05:00.181422 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:00.181369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6" podStartSLOduration=2.44788562 podStartE2EDuration="4.181352774s" totalImagesPullingTime="1.733467154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:00.180027388 +0000 UTC m=+387.182273741" watchObservedRunningTime="2026-09-10 21:05:00.181352774 +0000 UTC m=+387.183599190"
Sep 10 21:05:02.739133 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:02.739081    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"]
Sep 10 21:05:02.739530 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:02.739276    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw" podUID="d015bf2b-47bd-4e1d-9e42-2ffe951da02e" containerName="discovery" containerID="cri-o://7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9" gracePeriod=30
Sep 10 21:05:02.980488 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:02.980467    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:05:03.115054 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.114972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-kubeconfig\") pod \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") "
Sep 10 21:05:03.115054 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.115022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-local-certs\") pod \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") "
Sep 10 21:05:03.115054 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.115038    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-dns-cert\") pod \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") "
Sep 10 21:05:03.115346 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.115061    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nl62j\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-kube-api-access-nl62j\") pod \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") "
Sep 10 21:05:03.115346 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.115149    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-token\") pod \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") "
Sep 10 21:05:03.115346 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.115189    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-cacerts\") pod \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") "
Sep 10 21:05:03.115346 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.115221    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-ca-configmap\") pod \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\" (UID: \"d015bf2b-47bd-4e1d-9e42-2ffe951da02e\") "
Sep 10 21:05:03.115897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.115758    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d015bf2b-47bd-4e1d-9e42-2ffe951da02e" (UID: "d015bf2b-47bd-4e1d-9e42-2ffe951da02e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:05:03.117690 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.117658    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d015bf2b-47bd-4e1d-9e42-2ffe951da02e" (UID: "d015bf2b-47bd-4e1d-9e42-2ffe951da02e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:03.117817 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.117669    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-kube-api-access-nl62j" (OuterVolumeSpecName: "kube-api-access-nl62j") pod "d015bf2b-47bd-4e1d-9e42-2ffe951da02e" (UID: "d015bf2b-47bd-4e1d-9e42-2ffe951da02e"). InnerVolumeSpecName "kube-api-access-nl62j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:03.118002 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.117965    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d015bf2b-47bd-4e1d-9e42-2ffe951da02e" (UID: "d015bf2b-47bd-4e1d-9e42-2ffe951da02e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:03.118242 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.118214    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-cacerts" (OuterVolumeSpecName: "cacerts") pod "d015bf2b-47bd-4e1d-9e42-2ffe951da02e" (UID: "d015bf2b-47bd-4e1d-9e42-2ffe951da02e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:05:03.118342 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.118321    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-local-certs" (OuterVolumeSpecName: "local-certs") pod "d015bf2b-47bd-4e1d-9e42-2ffe951da02e" (UID: "d015bf2b-47bd-4e1d-9e42-2ffe951da02e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:03.118588 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.118561    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d015bf2b-47bd-4e1d-9e42-2ffe951da02e" (UID: "d015bf2b-47bd-4e1d-9e42-2ffe951da02e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:03.163961 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.163932    2580 generic.go:356] "Generic (PLEG): container finished" podID="d015bf2b-47bd-4e1d-9e42-2ffe951da02e" containerID="7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9" exitCode=0
Sep 10 21:05:03.164073 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.163969    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw" event={"ID":"d015bf2b-47bd-4e1d-9e42-2ffe951da02e","Type":"ContainerDied","Data":"7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9"}
Sep 10 21:05:03.164073 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.163977    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"
Sep 10 21:05:03.164073 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.164004    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw" event={"ID":"d015bf2b-47bd-4e1d-9e42-2ffe951da02e","Type":"ContainerDied","Data":"27477c0a36d1735f45177ee9f10260025fe80375760e94e5e0ca73fb20cde324"}
Sep 10 21:05:03.164073 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.164024    2580 scope.go:117] "RemoveContainer" containerID="7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9"
Sep 10 21:05:03.172840 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.172826    2580 scope.go:117] "RemoveContainer" containerID="7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9"
Sep 10 21:05:03.173099 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:05:03.173078    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9\": container with ID starting with 7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9 not found: ID does not exist" containerID="7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9"
Sep 10 21:05:03.173229 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.173106    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9"} err="failed to get container status \"7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9\": rpc error: code = NotFound desc = could not find container \"7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9\": container with ID starting with 7d7c1df84e3ed30a3315adf887f2e467b53c8b0e0020d51855a60a1b723977e9 not found: ID does not exist"
Sep 10 21:05:03.185130 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.185091    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"]
Sep 10 21:05:03.189408 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.189386    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-mpvxw"]
Sep 10 21:05:03.216603 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.216581    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:03.216603 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.216599    2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-cacerts\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:03.216732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.216607    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-ca-configmap\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:03.216732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.216617    2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-kubeconfig\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:03.216732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.216626    2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-local-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:03.216732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.216633    2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-istio-csr-dns-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:03.216732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.216642    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nl62j\" (UniqueName: \"kubernetes.io/projected/d015bf2b-47bd-4e1d-9e42-2ffe951da02e-kube-api-access-nl62j\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:03.517684 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:03.517654    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d015bf2b-47bd-4e1d-9e42-2ffe951da02e" path="/var/lib/kubelet/pods/d015bf2b-47bd-4e1d-9e42-2ffe951da02e/volumes"
Sep 10 21:05:11.156270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:11.156241    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-548fd67646-q8cd6"
Sep 10 21:05:36.368013 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.367975    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-554c9578b8-mvb89"]
Sep 10 21:05:36.368438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.368269    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d015bf2b-47bd-4e1d-9e42-2ffe951da02e" containerName="discovery"
Sep 10 21:05:36.368438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.368281    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d015bf2b-47bd-4e1d-9e42-2ffe951da02e" containerName="discovery"
Sep 10 21:05:36.368438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.368331    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d015bf2b-47bd-4e1d-9e42-2ffe951da02e" containerName="discovery"
Sep 10 21:05:36.371365 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.371350    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.373463 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.373438    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 21:05:36.374301 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.374283    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 21:05:36.374447 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.374419    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 21:05:36.374557 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.374512    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 21:05:36.374557 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.374522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 21:05:36.374676 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.374554    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 21:05:36.374676 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.374636    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 21:05:36.374862 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.374850    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7tkmb\""
Sep 10 21:05:36.378502 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.378176    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 21:05:36.382186 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.382166    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554c9578b8-mvb89"]
Sep 10 21:05:36.456424 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.456397    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-serving-cert\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.456424 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.456424    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-config\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.456573 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.456441    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-trusted-ca-bundle\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.456573 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.456490    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-oauth-config\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.456573 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.456523    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9fkn\" (UniqueName: \"kubernetes.io/projected/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-kube-api-access-g9fkn\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.456573 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.456550    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-service-ca\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.456697 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.456581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-oauth-serving-cert\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.557853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.557829    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-oauth-config\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558017 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.557871    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9fkn\" (UniqueName: \"kubernetes.io/projected/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-kube-api-access-g9fkn\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558017 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.557904    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-service-ca\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558017 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.557926    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-oauth-serving-cert\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558017 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.557977    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-serving-cert\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558017 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.557993    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-config\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558017 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.558007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-trusted-ca-bundle\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558791 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.558746    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-service-ca\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558886 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.558793    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-oauth-serving-cert\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558886 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.558811    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-config\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.558886 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.558882    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-trusted-ca-bundle\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.560561 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.560539    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-oauth-config\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.560640 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.560560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-console-serving-cert\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.565433 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.565413    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9fkn\" (UniqueName: \"kubernetes.io/projected/6a31ddc8-6c68-4dc5-9211-2eece856ccbc-kube-api-access-g9fkn\") pod \"console-554c9578b8-mvb89\" (UID: \"6a31ddc8-6c68-4dc5-9211-2eece856ccbc\") " pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.681247 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.681219    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:36.953197 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:36.953085    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-554c9578b8-mvb89"]
Sep 10 21:05:36.959516 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:05:36.959488    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a31ddc8_6c68_4dc5_9211_2eece856ccbc.slice/crio-5e8135530e2884db4f3342c73ac85b262ced49a69ac4a01bd98f1ffe21cddac6 WatchSource:0}: Error finding container 5e8135530e2884db4f3342c73ac85b262ced49a69ac4a01bd98f1ffe21cddac6: Status 404 returned error can't find the container with id 5e8135530e2884db4f3342c73ac85b262ced49a69ac4a01bd98f1ffe21cddac6
Sep 10 21:05:37.318241 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:37.318157    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554c9578b8-mvb89" event={"ID":"6a31ddc8-6c68-4dc5-9211-2eece856ccbc","Type":"ContainerStarted","Data":"ba10bdc6f069721aa3d6c9adb5340aa0baa451c3a190a68d2ab24ac2d067da78"}
Sep 10 21:05:37.318241 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:37.318209    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-554c9578b8-mvb89" event={"ID":"6a31ddc8-6c68-4dc5-9211-2eece856ccbc","Type":"ContainerStarted","Data":"5e8135530e2884db4f3342c73ac85b262ced49a69ac4a01bd98f1ffe21cddac6"}
Sep 10 21:05:37.338643 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:37.338582    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-554c9578b8-mvb89" podStartSLOduration=1.3385618959999999 podStartE2EDuration="1.338561896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:05:37.337442264 +0000 UTC m=+424.339688638" watchObservedRunningTime="2026-09-10 21:05:37.338561896 +0000 UTC m=+424.340808245"
Sep 10 21:05:46.681852 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:46.681818    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:46.682312 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:46.681872    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:46.686659 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:46.686635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:47.367340 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:47.367309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-554c9578b8-mvb89"
Sep 10 21:05:54.014829 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.014791    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"]
Sep 10 21:05:54.017524 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.017502    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.019887 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.019858    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 21:05:54.020521 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.020499    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 21:05:54.020595 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.020511    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-jvspl\""
Sep 10 21:05:54.026287 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.026252    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"]
Sep 10 21:05:54.092350 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.092321    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.092501 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.092363    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.092501 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.092392    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ffjf\" (UniqueName: \"kubernetes.io/projected/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-kube-api-access-6ffjf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.113678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.113648    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"]
Sep 10 21:05:54.115966 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.115950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.127653 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.127628    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"]
Sep 10 21:05:54.192760 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.192727    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.192914 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.192772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-276cp\" (UniqueName: \"kubernetes.io/projected/260e5483-9709-4c32-bd1c-78b301107c75-kube-api-access-276cp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.192914 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.192801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.192914 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.192831    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ffjf\" (UniqueName: \"kubernetes.io/projected/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-kube-api-access-6ffjf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.192914 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.192855    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.193068 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.192935    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.193219 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.193200    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.193259 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.193208    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.201615 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.201591    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ffjf\" (UniqueName: \"kubernetes.io/projected/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-kube-api-access-6ffjf\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.212978 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.212952    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"]
Sep 10 21:05:54.215638 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.215624    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.224612 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.224590    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"]
Sep 10 21:05:54.294344 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.294273    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.294344 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.294335    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-276cp\" (UniqueName: \"kubernetes.io/projected/260e5483-9709-4c32-bd1c-78b301107c75-kube-api-access-276cp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.294553 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.294363    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.294553 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.294381    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.294553 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.294402    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.294553 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.294421    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nfvn\" (UniqueName: \"kubernetes.io/projected/335999e5-3af3-4d62-9b61-0ec02f3975df-kube-api-access-7nfvn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.294800 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.294779    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.294800 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.294791    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.301728 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.301707    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-276cp\" (UniqueName: \"kubernetes.io/projected/260e5483-9709-4c32-bd1c-78b301107c75-kube-api-access-276cp\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.314651 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.314628    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"]
Sep 10 21:05:54.317095 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.317081    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.326030 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.326010    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"]
Sep 10 21:05:54.326940 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.326922    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:54.395487 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.395460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtrr\" (UniqueName: \"kubernetes.io/projected/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-kube-api-access-jwtrr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.395609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.395502    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.395609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.395548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nfvn\" (UniqueName: \"kubernetes.io/projected/335999e5-3af3-4d62-9b61-0ec02f3975df-kube-api-access-7nfvn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.395609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.395592    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.395734 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.395658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.395734 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.395675    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.395820 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.395800    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.396065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.396036    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.404566 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.404546    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nfvn\" (UniqueName: \"kubernetes.io/projected/335999e5-3af3-4d62-9b61-0ec02f3975df-kube-api-access-7nfvn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.424950 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.424913    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:54.496299 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.496229    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.496299 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.496261    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.496299 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.496281    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtrr\" (UniqueName: \"kubernetes.io/projected/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-kube-api-access-jwtrr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.496690 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.496669    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.496690 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.496683    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.504682 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.504659    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtrr\" (UniqueName: \"kubernetes.io/projected/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-kube-api-access-jwtrr\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.524810 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.524789    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:54.601286 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.601258    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"]
Sep 10 21:05:54.603515 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:05:54.603487    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99b56d4_cbce_461b_b6e4_8d2c7da28d94.slice/crio-697a12dc0df1e191b81480da5a84a1802852f42762d142f8bc43e03e3d6f2c78 WatchSource:0}: Error finding container 697a12dc0df1e191b81480da5a84a1802852f42762d142f8bc43e03e3d6f2c78: Status 404 returned error can't find the container with id 697a12dc0df1e191b81480da5a84a1802852f42762d142f8bc43e03e3d6f2c78
Sep 10 21:05:54.626911 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.626889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:54.699598 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.699574    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"]
Sep 10 21:05:54.701425 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:05:54.701395    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260e5483_9709_4c32_bd1c_78b301107c75.slice/crio-30cdecdc631c0d656efb6e9cd73200a203e37d6bdbded02ca3b95a805cdeb8b5 WatchSource:0}: Error finding container 30cdecdc631c0d656efb6e9cd73200a203e37d6bdbded02ca3b95a805cdeb8b5: Status 404 returned error can't find the container with id 30cdecdc631c0d656efb6e9cd73200a203e37d6bdbded02ca3b95a805cdeb8b5
Sep 10 21:05:54.794864 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.794845    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"]
Sep 10 21:05:54.796786 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:05:54.796764    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod335999e5_3af3_4d62_9b61_0ec02f3975df.slice/crio-b433e6e3709e44f901a01219c25cf88b0706266a579be67402203683b3e136bf WatchSource:0}: Error finding container b433e6e3709e44f901a01219c25cf88b0706266a579be67402203683b3e136bf: Status 404 returned error can't find the container with id b433e6e3709e44f901a01219c25cf88b0706266a579be67402203683b3e136bf
Sep 10 21:05:54.897449 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:54.897404    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"]
Sep 10 21:05:54.898882 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:05:54.898861    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94cbc4f0_122e_48cb_a3b9_02e45ca12b52.slice/crio-6a4c70a5c084126aad36f87c7e818ad7dc42418598f744bd9c4e21c54d86f360 WatchSource:0}: Error finding container 6a4c70a5c084126aad36f87c7e818ad7dc42418598f744bd9c4e21c54d86f360: Status 404 returned error can't find the container with id 6a4c70a5c084126aad36f87c7e818ad7dc42418598f744bd9c4e21c54d86f360
Sep 10 21:05:55.401390 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.401356    2580 generic.go:356] "Generic (PLEG): container finished" podID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerID="1fb3de0594fde621652b6f9714851f35a85ac8415e4a54dc5b97035274d9e1ec" exitCode=0
Sep 10 21:05:55.401855 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.401458    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj" event={"ID":"94cbc4f0-122e-48cb-a3b9-02e45ca12b52","Type":"ContainerDied","Data":"1fb3de0594fde621652b6f9714851f35a85ac8415e4a54dc5b97035274d9e1ec"}
Sep 10 21:05:55.401855 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.401526    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj" event={"ID":"94cbc4f0-122e-48cb-a3b9-02e45ca12b52","Type":"ContainerStarted","Data":"6a4c70a5c084126aad36f87c7e818ad7dc42418598f744bd9c4e21c54d86f360"}
Sep 10 21:05:55.403088 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.403012    2580 generic.go:356] "Generic (PLEG): container finished" podID="260e5483-9709-4c32-bd1c-78b301107c75" containerID="583f4fa7ee6818008eda0a8ddf717492ce07917a1c62a8fadf047fbc9b1db5e3" exitCode=0
Sep 10 21:05:55.403183 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.403091    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx" event={"ID":"260e5483-9709-4c32-bd1c-78b301107c75","Type":"ContainerDied","Data":"583f4fa7ee6818008eda0a8ddf717492ce07917a1c62a8fadf047fbc9b1db5e3"}
Sep 10 21:05:55.403183 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.403162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx" event={"ID":"260e5483-9709-4c32-bd1c-78b301107c75","Type":"ContainerStarted","Data":"30cdecdc631c0d656efb6e9cd73200a203e37d6bdbded02ca3b95a805cdeb8b5"}
Sep 10 21:05:55.404728 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.404708    2580 generic.go:356] "Generic (PLEG): container finished" podID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerID="f723cc854c6edd1520698ff55d3cc877f9f36f2bf63258b052ab333fa8c26717" exitCode=0
Sep 10 21:05:55.404827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.404796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd" event={"ID":"d99b56d4-cbce-461b-b6e4-8d2c7da28d94","Type":"ContainerDied","Data":"f723cc854c6edd1520698ff55d3cc877f9f36f2bf63258b052ab333fa8c26717"}
Sep 10 21:05:55.404890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.404842    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd" event={"ID":"d99b56d4-cbce-461b-b6e4-8d2c7da28d94","Type":"ContainerStarted","Data":"697a12dc0df1e191b81480da5a84a1802852f42762d142f8bc43e03e3d6f2c78"}
Sep 10 21:05:55.406630 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.406612    2580 generic.go:356] "Generic (PLEG): container finished" podID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerID="c630547f21cf53270223fab40a2be9826dc65f6e008f29e128f74f89837736a7" exitCode=0
Sep 10 21:05:55.406708 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.406637    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r" event={"ID":"335999e5-3af3-4d62-9b61-0ec02f3975df","Type":"ContainerDied","Data":"c630547f21cf53270223fab40a2be9826dc65f6e008f29e128f74f89837736a7"}
Sep 10 21:05:55.406708 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:55.406663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r" event={"ID":"335999e5-3af3-4d62-9b61-0ec02f3975df","Type":"ContainerStarted","Data":"b433e6e3709e44f901a01219c25cf88b0706266a579be67402203683b3e136bf"}
Sep 10 21:05:56.416566 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:56.416533    2580 generic.go:356] "Generic (PLEG): container finished" podID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerID="851bee5f0dbded4d7321369e4a17d02d6ce0c80be8ef9e6d2293b8a2eefc5b41" exitCode=0
Sep 10 21:05:56.416956 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:56.416617    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r" event={"ID":"335999e5-3af3-4d62-9b61-0ec02f3975df","Type":"ContainerDied","Data":"851bee5f0dbded4d7321369e4a17d02d6ce0c80be8ef9e6d2293b8a2eefc5b41"}
Sep 10 21:05:56.418718 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:56.418697    2580 generic.go:356] "Generic (PLEG): container finished" podID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerID="f7627aaeb3a496341731ecb9ee72b6df6139ca1c04ef2b62aef111414bb80163" exitCode=0
Sep 10 21:05:56.418820 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:56.418777    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj" event={"ID":"94cbc4f0-122e-48cb-a3b9-02e45ca12b52","Type":"ContainerDied","Data":"f7627aaeb3a496341731ecb9ee72b6df6139ca1c04ef2b62aef111414bb80163"}
Sep 10 21:05:57.425154 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:57.425101    2580 generic.go:356] "Generic (PLEG): container finished" podID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerID="fec1b180a854b925aaa2279c5f0a8278be8ae723f14b2a5a485d659a96489ef9" exitCode=0
Sep 10 21:05:57.425580 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:57.425191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r" event={"ID":"335999e5-3af3-4d62-9b61-0ec02f3975df","Type":"ContainerDied","Data":"fec1b180a854b925aaa2279c5f0a8278be8ae723f14b2a5a485d659a96489ef9"}
Sep 10 21:05:57.427279 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:57.427258    2580 generic.go:356] "Generic (PLEG): container finished" podID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerID="b8f810703203caf2069a0d786e50d91031cdeb392cf6304526e0d83873508fc5" exitCode=0
Sep 10 21:05:57.427392 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:57.427325    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj" event={"ID":"94cbc4f0-122e-48cb-a3b9-02e45ca12b52","Type":"ContainerDied","Data":"b8f810703203caf2069a0d786e50d91031cdeb392cf6304526e0d83873508fc5"}
Sep 10 21:05:57.429025 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:57.429005    2580 generic.go:356] "Generic (PLEG): container finished" podID="260e5483-9709-4c32-bd1c-78b301107c75" containerID="fa3273ddf74b8607afe46d87e87dced5f9436af68ec2959a2d819344dd530a0a" exitCode=0
Sep 10 21:05:57.429134 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:57.429095    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx" event={"ID":"260e5483-9709-4c32-bd1c-78b301107c75","Type":"ContainerDied","Data":"fa3273ddf74b8607afe46d87e87dced5f9436af68ec2959a2d819344dd530a0a"}
Sep 10 21:05:57.431044 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:57.431028    2580 generic.go:356] "Generic (PLEG): container finished" podID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerID="51dea8b1bea5e134b15c11886f9994faf2d37b842fb178138e991040d8596b16" exitCode=0
Sep 10 21:05:57.431174 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:57.431078    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd" event={"ID":"d99b56d4-cbce-461b-b6e4-8d2c7da28d94","Type":"ContainerDied","Data":"51dea8b1bea5e134b15c11886f9994faf2d37b842fb178138e991040d8596b16"}
Sep 10 21:05:58.438112 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.438080    2580 generic.go:356] "Generic (PLEG): container finished" podID="260e5483-9709-4c32-bd1c-78b301107c75" containerID="ebd421306cd3fc5d1e5cf57f0f94ff7eb7660346eff08645a92d0181fa816e31" exitCode=0
Sep 10 21:05:58.438548 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.438162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx" event={"ID":"260e5483-9709-4c32-bd1c-78b301107c75","Type":"ContainerDied","Data":"ebd421306cd3fc5d1e5cf57f0f94ff7eb7660346eff08645a92d0181fa816e31"}
Sep 10 21:05:58.440343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.440323    2580 generic.go:356] "Generic (PLEG): container finished" podID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerID="55ce9cbcc920707b6d21c4af25c4990d051242c845fa5d36041705c9a7bb3998" exitCode=0
Sep 10 21:05:58.440444 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.440385    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd" event={"ID":"d99b56d4-cbce-461b-b6e4-8d2c7da28d94","Type":"ContainerDied","Data":"55ce9cbcc920707b6d21c4af25c4990d051242c845fa5d36041705c9a7bb3998"}
Sep 10 21:05:58.567525 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.567503    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:58.596300 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.596276    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:58.630320 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.630291    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-util\") pod \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") "
Sep 10 21:05:58.630458 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.630332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwtrr\" (UniqueName: \"kubernetes.io/projected/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-kube-api-access-jwtrr\") pod \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") "
Sep 10 21:05:58.630458 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.630405    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-bundle\") pod \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\" (UID: \"94cbc4f0-122e-48cb-a3b9-02e45ca12b52\") "
Sep 10 21:05:58.630913 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.630887    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-bundle" (OuterVolumeSpecName: "bundle") pod "94cbc4f0-122e-48cb-a3b9-02e45ca12b52" (UID: "94cbc4f0-122e-48cb-a3b9-02e45ca12b52"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:58.632527 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.632506    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-kube-api-access-jwtrr" (OuterVolumeSpecName: "kube-api-access-jwtrr") pod "94cbc4f0-122e-48cb-a3b9-02e45ca12b52" (UID: "94cbc4f0-122e-48cb-a3b9-02e45ca12b52"). InnerVolumeSpecName "kube-api-access-jwtrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:58.635719 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.635680    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-util" (OuterVolumeSpecName: "util") pod "94cbc4f0-122e-48cb-a3b9-02e45ca12b52" (UID: "94cbc4f0-122e-48cb-a3b9-02e45ca12b52"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:58.731639 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.731575    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-bundle\") pod \"335999e5-3af3-4d62-9b61-0ec02f3975df\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") "
Sep 10 21:05:58.731639 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.731608    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-util\") pod \"335999e5-3af3-4d62-9b61-0ec02f3975df\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") "
Sep 10 21:05:58.731833 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.731643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7nfvn\" (UniqueName: \"kubernetes.io/projected/335999e5-3af3-4d62-9b61-0ec02f3975df-kube-api-access-7nfvn\") pod \"335999e5-3af3-4d62-9b61-0ec02f3975df\" (UID: \"335999e5-3af3-4d62-9b61-0ec02f3975df\") "
Sep 10 21:05:58.731833 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.731749    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.731833 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.731759    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwtrr\" (UniqueName: \"kubernetes.io/projected/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-kube-api-access-jwtrr\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.731833 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.731768    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94cbc4f0-122e-48cb-a3b9-02e45ca12b52-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.732165 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.732111    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-bundle" (OuterVolumeSpecName: "bundle") pod "335999e5-3af3-4d62-9b61-0ec02f3975df" (UID: "335999e5-3af3-4d62-9b61-0ec02f3975df"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:58.733834 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.733813    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/335999e5-3af3-4d62-9b61-0ec02f3975df-kube-api-access-7nfvn" (OuterVolumeSpecName: "kube-api-access-7nfvn") pod "335999e5-3af3-4d62-9b61-0ec02f3975df" (UID: "335999e5-3af3-4d62-9b61-0ec02f3975df"). InnerVolumeSpecName "kube-api-access-7nfvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:58.736926 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.736908    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-util" (OuterVolumeSpecName: "util") pod "335999e5-3af3-4d62-9b61-0ec02f3975df" (UID: "335999e5-3af3-4d62-9b61-0ec02f3975df"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:58.832947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.832923    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.832947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.832944    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/335999e5-3af3-4d62-9b61-0ec02f3975df-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:58.833074 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:58.832952    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nfvn\" (UniqueName: \"kubernetes.io/projected/335999e5-3af3-4d62-9b61-0ec02f3975df-kube-api-access-7nfvn\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:59.451300 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.451272    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj"
Sep 10 21:05:59.451764 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.451310    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c2nzkj" event={"ID":"94cbc4f0-122e-48cb-a3b9-02e45ca12b52","Type":"ContainerDied","Data":"6a4c70a5c084126aad36f87c7e818ad7dc42418598f744bd9c4e21c54d86f360"}
Sep 10 21:05:59.451764 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.451381    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a4c70a5c084126aad36f87c7e818ad7dc42418598f744bd9c4e21c54d86f360"
Sep 10 21:05:59.453270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.453249    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r"
Sep 10 21:05:59.453380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.453271    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kvw9r" event={"ID":"335999e5-3af3-4d62-9b61-0ec02f3975df","Type":"ContainerDied","Data":"b433e6e3709e44f901a01219c25cf88b0706266a579be67402203683b3e136bf"}
Sep 10 21:05:59.453380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.453304    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b433e6e3709e44f901a01219c25cf88b0706266a579be67402203683b3e136bf"
Sep 10 21:05:59.610298 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.610270    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:05:59.612274 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.612256    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:05:59.741310 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.741219    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-276cp\" (UniqueName: \"kubernetes.io/projected/260e5483-9709-4c32-bd1c-78b301107c75-kube-api-access-276cp\") pod \"260e5483-9709-4c32-bd1c-78b301107c75\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") "
Sep 10 21:05:59.741310 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.741265    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-bundle\") pod \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") "
Sep 10 21:05:59.741310 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.741290    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ffjf\" (UniqueName: \"kubernetes.io/projected/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-kube-api-access-6ffjf\") pod \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") "
Sep 10 21:05:59.741581 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.741328    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-util\") pod \"260e5483-9709-4c32-bd1c-78b301107c75\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") "
Sep 10 21:05:59.741581 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.741373    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-bundle\") pod \"260e5483-9709-4c32-bd1c-78b301107c75\" (UID: \"260e5483-9709-4c32-bd1c-78b301107c75\") "
Sep 10 21:05:59.741581 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.741387    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-util\") pod \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\" (UID: \"d99b56d4-cbce-461b-b6e4-8d2c7da28d94\") "
Sep 10 21:05:59.741913 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.741882    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-bundle" (OuterVolumeSpecName: "bundle") pod "260e5483-9709-4c32-bd1c-78b301107c75" (UID: "260e5483-9709-4c32-bd1c-78b301107c75"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:59.742087 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.742060    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-bundle" (OuterVolumeSpecName: "bundle") pod "d99b56d4-cbce-461b-b6e4-8d2c7da28d94" (UID: "d99b56d4-cbce-461b-b6e4-8d2c7da28d94"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:59.743588 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.743564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-kube-api-access-6ffjf" (OuterVolumeSpecName: "kube-api-access-6ffjf") pod "d99b56d4-cbce-461b-b6e4-8d2c7da28d94" (UID: "d99b56d4-cbce-461b-b6e4-8d2c7da28d94"). InnerVolumeSpecName "kube-api-access-6ffjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:59.743683 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.743612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260e5483-9709-4c32-bd1c-78b301107c75-kube-api-access-276cp" (OuterVolumeSpecName: "kube-api-access-276cp") pod "260e5483-9709-4c32-bd1c-78b301107c75" (UID: "260e5483-9709-4c32-bd1c-78b301107c75"). InnerVolumeSpecName "kube-api-access-276cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:05:59.746767 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.746748    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-util" (OuterVolumeSpecName: "util") pod "260e5483-9709-4c32-bd1c-78b301107c75" (UID: "260e5483-9709-4c32-bd1c-78b301107c75"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:59.747260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.747236    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-util" (OuterVolumeSpecName: "util") pod "d99b56d4-cbce-461b-b6e4-8d2c7da28d94" (UID: "d99b56d4-cbce-461b-b6e4-8d2c7da28d94"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:05:59.842752 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.842716    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:59.842752 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.842743    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:59.842752 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.842752    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-276cp\" (UniqueName: \"kubernetes.io/projected/260e5483-9709-4c32-bd1c-78b301107c75-kube-api-access-276cp\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:59.842752 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.842762    2580 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-bundle\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:59.842989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.842770    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ffjf\" (UniqueName: \"kubernetes.io/projected/d99b56d4-cbce-461b-b6e4-8d2c7da28d94-kube-api-access-6ffjf\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:05:59.842989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:05:59.842778    2580 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/260e5483-9709-4c32-bd1c-78b301107c75-util\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:06:00.459809 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:00.459777    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx"
Sep 10 21:06:00.459809 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:00.459796    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6mmddx" event={"ID":"260e5483-9709-4c32-bd1c-78b301107c75","Type":"ContainerDied","Data":"30cdecdc631c0d656efb6e9cd73200a203e37d6bdbded02ca3b95a805cdeb8b5"}
Sep 10 21:06:00.460262 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:00.459878    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30cdecdc631c0d656efb6e9cd73200a203e37d6bdbded02ca3b95a805cdeb8b5"
Sep 10 21:06:00.461698 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:00.461678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd" event={"ID":"d99b56d4-cbce-461b-b6e4-8d2c7da28d94","Type":"ContainerDied","Data":"697a12dc0df1e191b81480da5a84a1802852f42762d142f8bc43e03e3d6f2c78"}
Sep 10 21:06:00.461814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:00.461714    2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697a12dc0df1e191b81480da5a84a1802852f42762d142f8bc43e03e3d6f2c78"
Sep 10 21:06:00.461814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:00.461717    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a9g6zd"
Sep 10 21:06:06.474437 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474400    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"]
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474689    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerName="util"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474699    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerName="util"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474707    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerName="extract"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474713    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerName="extract"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474719    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerName="extract"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474725    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerName="extract"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474732    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260e5483-9709-4c32-bd1c-78b301107c75" containerName="extract"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474738    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="260e5483-9709-4c32-bd1c-78b301107c75" containerName="extract"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474745    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerName="util"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474751    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerName="util"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474756    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260e5483-9709-4c32-bd1c-78b301107c75" containerName="pull"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474762    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="260e5483-9709-4c32-bd1c-78b301107c75" containerName="pull"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474768    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerName="util"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474772    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerName="util"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474777    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="260e5483-9709-4c32-bd1c-78b301107c75" containerName="util"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474782    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="260e5483-9709-4c32-bd1c-78b301107c75" containerName="util"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474787    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerName="pull"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474792    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerName="pull"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474796    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerName="extract"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474801    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerName="extract"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474808    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerName="pull"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474813    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerName="pull"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474823    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerName="pull"
Sep 10 21:06:06.474831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474827    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerName="pull"
Sep 10 21:06:06.475578 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474871    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="260e5483-9709-4c32-bd1c-78b301107c75" containerName="extract"
Sep 10 21:06:06.475578 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474879    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="335999e5-3af3-4d62-9b61-0ec02f3975df" containerName="extract"
Sep 10 21:06:06.475578 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474886    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="94cbc4f0-122e-48cb-a3b9-02e45ca12b52" containerName="extract"
Sep 10 21:06:06.475578 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.474892    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="d99b56d4-cbce-461b-b6e4-8d2c7da28d94" containerName="extract"
Sep 10 21:06:06.477874 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.477856    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:06.480327 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.480307    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tqkjj\""
Sep 10 21:06:06.480415 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.480371    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 21:06:06.480939 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.480925    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 21:06:06.489060 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.489038    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"]
Sep 10 21:06:06.601963 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.601927    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f9mk\" (UniqueName: \"kubernetes.io/projected/17b543aa-8ce0-4e33-b03a-3c63946b0a49-kube-api-access-6f9mk\") pod \"limitador-operator-controller-manager-74495c69f8-kj77l\" (UID: \"17b543aa-8ce0-4e33-b03a-3c63946b0a49\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:06.703024 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.702979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f9mk\" (UniqueName: \"kubernetes.io/projected/17b543aa-8ce0-4e33-b03a-3c63946b0a49-kube-api-access-6f9mk\") pod \"limitador-operator-controller-manager-74495c69f8-kj77l\" (UID: \"17b543aa-8ce0-4e33-b03a-3c63946b0a49\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:06.720710 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.720676    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f9mk\" (UniqueName: \"kubernetes.io/projected/17b543aa-8ce0-4e33-b03a-3c63946b0a49-kube-api-access-6f9mk\") pod \"limitador-operator-controller-manager-74495c69f8-kj77l\" (UID: \"17b543aa-8ce0-4e33-b03a-3c63946b0a49\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:06.788597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:06.788516    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:07.074346 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:07.074320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"]
Sep 10 21:06:07.075578 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:06:07.075545    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17b543aa_8ce0_4e33_b03a_3c63946b0a49.slice/crio-c2338aafc80a82573db1f1e8d32debea229921ed69cf81d40d71001d7bfc615c WatchSource:0}: Error finding container c2338aafc80a82573db1f1e8d32debea229921ed69cf81d40d71001d7bfc615c: Status 404 returned error can't find the container with id c2338aafc80a82573db1f1e8d32debea229921ed69cf81d40d71001d7bfc615c
Sep 10 21:06:07.495044 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:07.495009    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l" event={"ID":"17b543aa-8ce0-4e33-b03a-3c63946b0a49","Type":"ContainerStarted","Data":"c2338aafc80a82573db1f1e8d32debea229921ed69cf81d40d71001d7bfc615c"}
Sep 10 21:06:09.506718 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:09.506679    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l" event={"ID":"17b543aa-8ce0-4e33-b03a-3c63946b0a49","Type":"ContainerStarted","Data":"7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac"}
Sep 10 21:06:09.507141 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:09.506804    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:09.534226 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:09.534180    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l" podStartSLOduration=1.259113049 podStartE2EDuration="3.534164301s" totalImagesPullingTime="2.275051252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:09.531527125 +0000 UTC m=+456.533773472" watchObservedRunningTime="2026-09-10 21:06:09.534164301 +0000 UTC m=+456.536410648"
Sep 10 21:06:13.316587 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.316553    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"]
Sep 10 21:06:13.319132 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.319096    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:13.321180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.321158    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9dkn7\""
Sep 10 21:06:13.333924 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.333903    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"]
Sep 10 21:06:13.459268 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.459219    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28e4ccfc-abba-4e6f-927f-37d219d00d5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" (UID: \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:13.459268 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.459281    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjvdd\" (UniqueName: \"kubernetes.io/projected/28e4ccfc-abba-4e6f-927f-37d219d00d5f-kube-api-access-mjvdd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" (UID: \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:13.559790 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.559756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28e4ccfc-abba-4e6f-927f-37d219d00d5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" (UID: \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:13.559967 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.559808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjvdd\" (UniqueName: \"kubernetes.io/projected/28e4ccfc-abba-4e6f-927f-37d219d00d5f-kube-api-access-mjvdd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" (UID: \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:13.560115 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.560093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28e4ccfc-abba-4e6f-927f-37d219d00d5f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" (UID: \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:13.573698 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.573646    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjvdd\" (UniqueName: \"kubernetes.io/projected/28e4ccfc-abba-4e6f-927f-37d219d00d5f-kube-api-access-mjvdd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" (UID: \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:13.628551 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.628528    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:13.903178 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:13.903154    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"]
Sep 10 21:06:13.905476 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:06:13.905449    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e4ccfc_abba_4e6f_927f_37d219d00d5f.slice/crio-aa90c6fd842c82ac4928f68672e8771f448977423f501e8e836f42e4bbc80ef0 WatchSource:0}: Error finding container aa90c6fd842c82ac4928f68672e8771f448977423f501e8e836f42e4bbc80ef0: Status 404 returned error can't find the container with id aa90c6fd842c82ac4928f68672e8771f448977423f501e8e836f42e4bbc80ef0
Sep 10 21:06:14.530624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:14.530589    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf" event={"ID":"28e4ccfc-abba-4e6f-927f-37d219d00d5f","Type":"ContainerStarted","Data":"aa90c6fd842c82ac4928f68672e8771f448977423f501e8e836f42e4bbc80ef0"}
Sep 10 21:06:15.526019 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.525982    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"]
Sep 10 21:06:15.528319 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.528301    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"
Sep 10 21:06:15.530921 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.530899    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-m5mq9\""
Sep 10 21:06:15.531356 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.530937    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 21:06:15.539565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.539543    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"]
Sep 10 21:06:15.677142 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.677082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhxd\" (UniqueName: \"kubernetes.io/projected/a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77-kube-api-access-fvhxd\") pod \"dns-operator-controller-manager-7b84cd7c9c-g2wsv\" (UID: \"a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"
Sep 10 21:06:15.777813 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.777737    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhxd\" (UniqueName: \"kubernetes.io/projected/a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77-kube-api-access-fvhxd\") pod \"dns-operator-controller-manager-7b84cd7c9c-g2wsv\" (UID: \"a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"
Sep 10 21:06:15.790403 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.790336    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhxd\" (UniqueName: \"kubernetes.io/projected/a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77-kube-api-access-fvhxd\") pod \"dns-operator-controller-manager-7b84cd7c9c-g2wsv\" (UID: \"a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"
Sep 10 21:06:15.841659 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:15.841629    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"
Sep 10 21:06:16.148016 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:16.147987    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"]
Sep 10 21:06:16.148199 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:06:16.148175    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4bd7e78_6c5f_47bd_8ea7_d2696b0f1d77.slice/crio-e3c8eb8551e5f6506c98ead4c0e90e77293b927c2b877996e42f21806fca1f01 WatchSource:0}: Error finding container e3c8eb8551e5f6506c98ead4c0e90e77293b927c2b877996e42f21806fca1f01: Status 404 returned error can't find the container with id e3c8eb8551e5f6506c98ead4c0e90e77293b927c2b877996e42f21806fca1f01
Sep 10 21:06:16.543355 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:16.543268    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv" event={"ID":"a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77","Type":"ContainerStarted","Data":"e3c8eb8551e5f6506c98ead4c0e90e77293b927c2b877996e42f21806fca1f01"}
Sep 10 21:06:19.561839 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:19.561801    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf" event={"ID":"28e4ccfc-abba-4e6f-927f-37d219d00d5f","Type":"ContainerStarted","Data":"8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea"}
Sep 10 21:06:19.562299 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:19.561979    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:19.563417 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:19.563396    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv" event={"ID":"a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77","Type":"ContainerStarted","Data":"4a527b8c1265a33e565ec4f9106ae4ad4451ee242cfdb4e8cab8a81f72b579f0"}
Sep 10 21:06:19.563521 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:19.563474    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"
Sep 10 21:06:19.582369 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:19.582325    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf" podStartSLOduration=1.681367471 podStartE2EDuration="6.582311335s" totalImagesPullingTime="4.900943864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:19.580082452 +0000 UTC m=+466.582328799" watchObservedRunningTime="2026-09-10 21:06:19.582311335 +0000 UTC m=+466.584557684"
Sep 10 21:06:19.598206 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:19.598073    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv" podStartSLOduration=1.949436929 podStartE2EDuration="4.598055432s" totalImagesPullingTime="2.648618503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:19.594659774 +0000 UTC m=+466.596906120" watchObservedRunningTime="2026-09-10 21:06:19.598055432 +0000 UTC m=+466.600301779"
Sep 10 21:06:20.514278 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:20.514250    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:30.570902 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:30.570873    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:30.571364 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:30.570941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-g2wsv"
Sep 10 21:06:32.959623 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:32.959583    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"]
Sep 10 21:06:32.960094 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:32.960000    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" containerName="manager" containerID="cri-o://8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea" gracePeriod=2
Sep 10 21:06:32.986973 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:32.986945    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"]
Sep 10 21:06:32.995002 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:32.994961    2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"28e4ccfc-abba-4e6f-927f-37d219d00d5f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\": pods \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:32.997709 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:32.997669    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"]
Sep 10 21:06:32.998158 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:32.998109    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" containerName="manager"
Sep 10 21:06:32.998270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:32.998161    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" containerName="manager"
Sep 10 21:06:32.998270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:32.998255    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" containerName="manager"
Sep 10 21:06:33.000529 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.000511    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:33.001993 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.001973    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"]
Sep 10 21:06:33.002188 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.002153    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" containerName="manager" containerID="cri-o://7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac" gracePeriod=2
Sep 10 21:06:33.008272 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.008252    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"]
Sep 10 21:06:33.016881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.016859    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"]
Sep 10 21:06:33.026436 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.026414    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"]
Sep 10 21:06:33.026868 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.026853    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" containerName="manager"
Sep 10 21:06:33.026868 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.026870    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" containerName="manager"
Sep 10 21:06:33.026964 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.026928    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" containerName="manager"
Sep 10 21:06:33.029102 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.029083    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"
Sep 10 21:06:33.033258 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.033228    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:33.035037 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.035009    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:33.040283 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.040260    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"]
Sep 10 21:06:33.054376 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.054344    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj77l\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:33.105462 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.105436    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwgx7\" (UniqueName: \"kubernetes.io/projected/32f1a730-590d-4fe9-8d34-1189d37629aa-kube-api-access-xwgx7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s856f\" (UID: \"32f1a730-590d-4fe9-8d34-1189d37629aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:33.105569 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.105476    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32f1a730-590d-4fe9-8d34-1189d37629aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s856f\" (UID: \"32f1a730-590d-4fe9-8d34-1189d37629aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:33.105569 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.105551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86rc\" (UniqueName: \"kubernetes.io/projected/9fb7cd0c-d3c9-4532-9612-aaff905a9b0e-kube-api-access-j86rc\") pod \"limitador-operator-controller-manager-74495c69f8-kxlwp\" (UID: \"9fb7cd0c-d3c9-4532-9612-aaff905a9b0e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"
Sep 10 21:06:33.206762 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.206725    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86rc\" (UniqueName: \"kubernetes.io/projected/9fb7cd0c-d3c9-4532-9612-aaff905a9b0e-kube-api-access-j86rc\") pod \"limitador-operator-controller-manager-74495c69f8-kxlwp\" (UID: \"9fb7cd0c-d3c9-4532-9612-aaff905a9b0e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"
Sep 10 21:06:33.206931 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.206776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwgx7\" (UniqueName: \"kubernetes.io/projected/32f1a730-590d-4fe9-8d34-1189d37629aa-kube-api-access-xwgx7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s856f\" (UID: \"32f1a730-590d-4fe9-8d34-1189d37629aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:33.206931 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.206804    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32f1a730-590d-4fe9-8d34-1189d37629aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s856f\" (UID: \"32f1a730-590d-4fe9-8d34-1189d37629aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:33.207168 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.207149    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32f1a730-590d-4fe9-8d34-1189d37629aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s856f\" (UID: \"32f1a730-590d-4fe9-8d34-1189d37629aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:33.221854 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.221835    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86rc\" (UniqueName: \"kubernetes.io/projected/9fb7cd0c-d3c9-4532-9612-aaff905a9b0e-kube-api-access-j86rc\") pod \"limitador-operator-controller-manager-74495c69f8-kxlwp\" (UID: \"9fb7cd0c-d3c9-4532-9612-aaff905a9b0e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"
Sep 10 21:06:33.221947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.221910    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwgx7\" (UniqueName: \"kubernetes.io/projected/32f1a730-590d-4fe9-8d34-1189d37629aa-kube-api-access-xwgx7\") pod \"kuadrant-operator-controller-manager-76f8f686bf-s856f\" (UID: \"32f1a730-590d-4fe9-8d34-1189d37629aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:33.238762 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.238737    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:33.240909 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.240887    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj77l\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:33.241321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.241304    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:33.242279 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.242259    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:33.243965 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.243942    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj77l\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:33.245433 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.245415    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:33.308045 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.308019    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f9mk\" (UniqueName: \"kubernetes.io/projected/17b543aa-8ce0-4e33-b03a-3c63946b0a49-kube-api-access-6f9mk\") pod \"17b543aa-8ce0-4e33-b03a-3c63946b0a49\" (UID: \"17b543aa-8ce0-4e33-b03a-3c63946b0a49\") "
Sep 10 21:06:33.308180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.308053    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjvdd\" (UniqueName: \"kubernetes.io/projected/28e4ccfc-abba-4e6f-927f-37d219d00d5f-kube-api-access-mjvdd\") pod \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\" (UID: \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\") "
Sep 10 21:06:33.308180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.308111    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28e4ccfc-abba-4e6f-927f-37d219d00d5f-extensions-socket-volume\") pod \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\" (UID: \"28e4ccfc-abba-4e6f-927f-37d219d00d5f\") "
Sep 10 21:06:33.308644 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.308622    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e4ccfc-abba-4e6f-927f-37d219d00d5f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "28e4ccfc-abba-4e6f-927f-37d219d00d5f" (UID: "28e4ccfc-abba-4e6f-927f-37d219d00d5f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:06:33.310350 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.310327    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17b543aa-8ce0-4e33-b03a-3c63946b0a49-kube-api-access-6f9mk" (OuterVolumeSpecName: "kube-api-access-6f9mk") pod "17b543aa-8ce0-4e33-b03a-3c63946b0a49" (UID: "17b543aa-8ce0-4e33-b03a-3c63946b0a49"). InnerVolumeSpecName "kube-api-access-6f9mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:06:33.310449 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.310368    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e4ccfc-abba-4e6f-927f-37d219d00d5f-kube-api-access-mjvdd" (OuterVolumeSpecName: "kube-api-access-mjvdd") pod "28e4ccfc-abba-4e6f-927f-37d219d00d5f" (UID: "28e4ccfc-abba-4e6f-927f-37d219d00d5f"). InnerVolumeSpecName "kube-api-access-mjvdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:06:33.383348 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.383325    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:33.389215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.389198    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"
Sep 10 21:06:33.408854 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.408831    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f9mk\" (UniqueName: \"kubernetes.io/projected/17b543aa-8ce0-4e33-b03a-3c63946b0a49-kube-api-access-6f9mk\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:06:33.408854 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.408854    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjvdd\" (UniqueName: \"kubernetes.io/projected/28e4ccfc-abba-4e6f-927f-37d219d00d5f-kube-api-access-mjvdd\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:06:33.408988 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.408864    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/28e4ccfc-abba-4e6f-927f-37d219d00d5f-extensions-socket-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:06:33.518531 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.518452    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-kj77l\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:33.521666 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.520444    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" path="/var/lib/kubelet/pods/17b543aa-8ce0-4e33-b03a-3c63946b0a49/volumes"
Sep 10 21:06:33.521666 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.520922    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" path="/var/lib/kubelet/pods/28e4ccfc-abba-4e6f-927f-37d219d00d5f/volumes"
Sep 10 21:06:33.522065 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.522030    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-wf6wf\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:33.633464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.633429    2580 generic.go:356] "Generic (PLEG): container finished" podID="28e4ccfc-abba-4e6f-927f-37d219d00d5f" containerID="8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea" exitCode=0
Sep 10 21:06:33.633632 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.633475    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-wf6wf"
Sep 10 21:06:33.633632 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.633491    2580 scope.go:117] "RemoveContainer" containerID="8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea"
Sep 10 21:06:33.634931 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.634909    2580 generic.go:356] "Generic (PLEG): container finished" podID="17b543aa-8ce0-4e33-b03a-3c63946b0a49" containerID="7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac" exitCode=0
Sep 10 21:06:33.635027 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.634952    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kj77l"
Sep 10 21:06:33.642352 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.642333    2580 scope.go:117] "RemoveContainer" containerID="8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea"
Sep 10 21:06:33.642666 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.642647    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea\": container with ID starting with 8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea not found: ID does not exist" containerID="8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea"
Sep 10 21:06:33.642751 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.642677    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea"} err="failed to get container status \"8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea\": rpc error: code = NotFound desc = could not find container \"8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea\": container with ID starting with 8cff458354643e145bf81a004a50bb8a1cd901e9ee444710e52392481220bbea not found: ID does not exist"
Sep 10 21:06:33.642751 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.642700    2580 scope.go:117] "RemoveContainer" containerID="7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac"
Sep 10 21:06:33.650601 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.650585    2580 scope.go:117] "RemoveContainer" containerID="7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac"
Sep 10 21:06:33.650858 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:33.650843    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac\": container with ID starting with 7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac not found: ID does not exist" containerID="7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac"
Sep 10 21:06:33.650897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.650864    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac"} err="failed to get container status \"7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac\": rpc error: code = NotFound desc = could not find container \"7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac\": container with ID starting with 7464ba8cf8bc176954645bfd79080d46a3d8d6fcae95332859cd849b9700c0ac not found: ID does not exist"
Sep 10 21:06:33.665037 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.665014    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"]
Sep 10 21:06:33.667982 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:06:33.667958    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f1a730_590d_4fe9_8d34_1189d37629aa.slice/crio-572215dcc0c0915460a6bcdde881a98a208da6423de9d5c8dc80a5a6cd28dbaf WatchSource:0}: Error finding container 572215dcc0c0915460a6bcdde881a98a208da6423de9d5c8dc80a5a6cd28dbaf: Status 404 returned error can't find the container with id 572215dcc0c0915460a6bcdde881a98a208da6423de9d5c8dc80a5a6cd28dbaf
Sep 10 21:06:33.676417 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:33.676397    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"]
Sep 10 21:06:33.679278 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:06:33.679260    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fb7cd0c_d3c9_4532_9612_aaff905a9b0e.slice/crio-430632bdd2629d246e81cab445c89e328f76010ffab34b26c30bceefd59d977a WatchSource:0}: Error finding container 430632bdd2629d246e81cab445c89e328f76010ffab34b26c30bceefd59d977a: Status 404 returned error can't find the container with id 430632bdd2629d246e81cab445c89e328f76010ffab34b26c30bceefd59d977a
Sep 10 21:06:34.643103 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:34.643068    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp" event={"ID":"9fb7cd0c-d3c9-4532-9612-aaff905a9b0e","Type":"ContainerStarted","Data":"5523cda3eda9606cf2edac87a68635ef72f64ba72831d2c20f063169ecc8054d"}
Sep 10 21:06:34.643545 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:34.643173    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp" event={"ID":"9fb7cd0c-d3c9-4532-9612-aaff905a9b0e","Type":"ContainerStarted","Data":"430632bdd2629d246e81cab445c89e328f76010ffab34b26c30bceefd59d977a"}
Sep 10 21:06:34.643545 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:34.643205    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"
Sep 10 21:06:34.644791 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:34.644767    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f" event={"ID":"32f1a730-590d-4fe9-8d34-1189d37629aa","Type":"ContainerStarted","Data":"cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889"}
Sep 10 21:06:34.644920 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:34.644811    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f" event={"ID":"32f1a730-590d-4fe9-8d34-1189d37629aa","Type":"ContainerStarted","Data":"572215dcc0c0915460a6bcdde881a98a208da6423de9d5c8dc80a5a6cd28dbaf"}
Sep 10 21:06:34.644920 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:34.644854    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:34.671348 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:34.671299    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp" podStartSLOduration=1.671285079 podStartE2EDuration="1.671285079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:34.671215491 +0000 UTC m=+481.673461840" watchObservedRunningTime="2026-09-10 21:06:34.671285079 +0000 UTC m=+481.673531427"
Sep 10 21:06:34.708525 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:34.708479    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f" podStartSLOduration=2.708465151 podStartE2EDuration="2.708465151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:34.70649876 +0000 UTC m=+481.708745108" watchObservedRunningTime="2026-09-10 21:06:34.708465151 +0000 UTC m=+481.710711497"
Sep 10 21:06:45.653796 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:45.653762    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-kxlwp"
Sep 10 21:06:45.654277 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:45.653839    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:48.203430 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.203399    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"]
Sep 10 21:06:48.203780 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.203597    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f" podUID="32f1a730-590d-4fe9-8d34-1189d37629aa" containerName="manager" containerID="cri-o://cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889" gracePeriod=10
Sep 10 21:06:48.440029 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.440003    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:48.486302 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.486233    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"]
Sep 10 21:06:48.486533 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.486518    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32f1a730-590d-4fe9-8d34-1189d37629aa" containerName="manager"
Sep 10 21:06:48.486533 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.486532    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="32f1a730-590d-4fe9-8d34-1189d37629aa" containerName="manager"
Sep 10 21:06:48.486677 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.486600    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="32f1a730-590d-4fe9-8d34-1189d37629aa" containerName="manager"
Sep 10 21:06:48.488658 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.488644    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:48.502648 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.502626    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"]
Sep 10 21:06:48.532748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.532723    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32f1a730-590d-4fe9-8d34-1189d37629aa-extensions-socket-volume\") pod \"32f1a730-590d-4fe9-8d34-1189d37629aa\" (UID: \"32f1a730-590d-4fe9-8d34-1189d37629aa\") "
Sep 10 21:06:48.532839 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.532752    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwgx7\" (UniqueName: \"kubernetes.io/projected/32f1a730-590d-4fe9-8d34-1189d37629aa-kube-api-access-xwgx7\") pod \"32f1a730-590d-4fe9-8d34-1189d37629aa\" (UID: \"32f1a730-590d-4fe9-8d34-1189d37629aa\") "
Sep 10 21:06:48.532900 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.532859    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c70afef-d251-4df2-ac24-595bf202ff6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxtfp\" (UID: \"5c70afef-d251-4df2-ac24-595bf202ff6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:48.532900 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.532875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl79j\" (UniqueName: \"kubernetes.io/projected/5c70afef-d251-4df2-ac24-595bf202ff6d-kube-api-access-kl79j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxtfp\" (UID: \"5c70afef-d251-4df2-ac24-595bf202ff6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:48.533087 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.533062    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f1a730-590d-4fe9-8d34-1189d37629aa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "32f1a730-590d-4fe9-8d34-1189d37629aa" (UID: "32f1a730-590d-4fe9-8d34-1189d37629aa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:06:48.535014 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.534991    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f1a730-590d-4fe9-8d34-1189d37629aa-kube-api-access-xwgx7" (OuterVolumeSpecName: "kube-api-access-xwgx7") pod "32f1a730-590d-4fe9-8d34-1189d37629aa" (UID: "32f1a730-590d-4fe9-8d34-1189d37629aa"). InnerVolumeSpecName "kube-api-access-xwgx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:06:48.634115 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.634079    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c70afef-d251-4df2-ac24-595bf202ff6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxtfp\" (UID: \"5c70afef-d251-4df2-ac24-595bf202ff6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:48.634115 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.634112    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl79j\" (UniqueName: \"kubernetes.io/projected/5c70afef-d251-4df2-ac24-595bf202ff6d-kube-api-access-kl79j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxtfp\" (UID: \"5c70afef-d251-4df2-ac24-595bf202ff6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:48.634374 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.634201    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/32f1a730-590d-4fe9-8d34-1189d37629aa-extensions-socket-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:06:48.634374 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.634216    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwgx7\" (UniqueName: \"kubernetes.io/projected/32f1a730-590d-4fe9-8d34-1189d37629aa-kube-api-access-xwgx7\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:06:48.634582 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.634559    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c70afef-d251-4df2-ac24-595bf202ff6d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxtfp\" (UID: \"5c70afef-d251-4df2-ac24-595bf202ff6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:48.642534 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.642511    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl79j\" (UniqueName: \"kubernetes.io/projected/5c70afef-d251-4df2-ac24-595bf202ff6d-kube-api-access-kl79j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-bxtfp\" (UID: \"5c70afef-d251-4df2-ac24-595bf202ff6d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:48.718202 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.718169    2580 generic.go:356] "Generic (PLEG): container finished" podID="32f1a730-590d-4fe9-8d34-1189d37629aa" containerID="cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889" exitCode=0
Sep 10 21:06:48.718349 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.718215    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"
Sep 10 21:06:48.718349 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.718255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f" event={"ID":"32f1a730-590d-4fe9-8d34-1189d37629aa","Type":"ContainerDied","Data":"cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889"}
Sep 10 21:06:48.718349 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.718303    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f" event={"ID":"32f1a730-590d-4fe9-8d34-1189d37629aa","Type":"ContainerDied","Data":"572215dcc0c0915460a6bcdde881a98a208da6423de9d5c8dc80a5a6cd28dbaf"}
Sep 10 21:06:48.718349 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.718342    2580 scope.go:117] "RemoveContainer" containerID="cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889"
Sep 10 21:06:48.727450 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.727434    2580 scope.go:117] "RemoveContainer" containerID="cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889"
Sep 10 21:06:48.727703 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:06:48.727687    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889\": container with ID starting with cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889 not found: ID does not exist" containerID="cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889"
Sep 10 21:06:48.727750 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.727708    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889"} err="failed to get container status \"cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889\": rpc error: code = NotFound desc = could not find container \"cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889\": container with ID starting with cb7825c2e02b1869a12fae9ef2a97a7bb2b3d026f083b94ff965a823772d9889 not found: ID does not exist"
Sep 10 21:06:48.747186 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.747134    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"]
Sep 10 21:06:48.753044 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.753024    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-s856f"]
Sep 10 21:06:48.798301 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:48.798283    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:49.078470 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:49.078442    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"]
Sep 10 21:06:49.080924 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:06:49.080889    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c70afef_d251_4df2_ac24_595bf202ff6d.slice/crio-d40a71e42f6d7144a30cce5317bf842a36e1d9f3a9f2c31ccb8682e42e45ffc9 WatchSource:0}: Error finding container d40a71e42f6d7144a30cce5317bf842a36e1d9f3a9f2c31ccb8682e42e45ffc9: Status 404 returned error can't find the container with id d40a71e42f6d7144a30cce5317bf842a36e1d9f3a9f2c31ccb8682e42e45ffc9
Sep 10 21:06:49.518612 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:49.518580    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32f1a730-590d-4fe9-8d34-1189d37629aa" path="/var/lib/kubelet/pods/32f1a730-590d-4fe9-8d34-1189d37629aa/volumes"
Sep 10 21:06:49.726448 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:49.726406    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp" event={"ID":"5c70afef-d251-4df2-ac24-595bf202ff6d","Type":"ContainerStarted","Data":"00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869"}
Sep 10 21:06:49.726609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:49.726464    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp" event={"ID":"5c70afef-d251-4df2-ac24-595bf202ff6d","Type":"ContainerStarted","Data":"d40a71e42f6d7144a30cce5317bf842a36e1d9f3a9f2c31ccb8682e42e45ffc9"}
Sep 10 21:06:49.726609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:49.726526    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:06:49.759231 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:06:49.759187    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp" podStartSLOduration=1.759171064 podStartE2EDuration="1.759171064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:06:49.757206285 +0000 UTC m=+496.759452631" watchObservedRunningTime="2026-09-10 21:06:49.759171064 +0000 UTC m=+496.761417410"
Sep 10 21:07:00.733718 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:00.733678    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:07:21.377534 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.377500    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2jjsb"]
Sep 10 21:07:21.380004 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.379988    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:21.382416 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.382395    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cn77b\""
Sep 10 21:07:21.382546 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.382517    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 21:07:21.392336 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.392315    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2jjsb"]
Sep 10 21:07:21.403277 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.403254    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/928fb574-d543-486a-9d5b-ce3d48609cbb-config-file\") pod \"limitador-limitador-8588749977-2jjsb\" (UID: \"928fb574-d543-486a-9d5b-ce3d48609cbb\") " pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:21.403362 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.403288    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pdxl\" (UniqueName: \"kubernetes.io/projected/928fb574-d543-486a-9d5b-ce3d48609cbb-kube-api-access-5pdxl\") pod \"limitador-limitador-8588749977-2jjsb\" (UID: \"928fb574-d543-486a-9d5b-ce3d48609cbb\") " pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:21.486926 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.486895    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2jjsb"]
Sep 10 21:07:21.503806 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.503776    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/928fb574-d543-486a-9d5b-ce3d48609cbb-config-file\") pod \"limitador-limitador-8588749977-2jjsb\" (UID: \"928fb574-d543-486a-9d5b-ce3d48609cbb\") " pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:21.503806 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.503809    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pdxl\" (UniqueName: \"kubernetes.io/projected/928fb574-d543-486a-9d5b-ce3d48609cbb-kube-api-access-5pdxl\") pod \"limitador-limitador-8588749977-2jjsb\" (UID: \"928fb574-d543-486a-9d5b-ce3d48609cbb\") " pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:21.504584 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.504564    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/928fb574-d543-486a-9d5b-ce3d48609cbb-config-file\") pod \"limitador-limitador-8588749977-2jjsb\" (UID: \"928fb574-d543-486a-9d5b-ce3d48609cbb\") " pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:21.517780 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.517752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pdxl\" (UniqueName: \"kubernetes.io/projected/928fb574-d543-486a-9d5b-ce3d48609cbb-kube-api-access-5pdxl\") pod \"limitador-limitador-8588749977-2jjsb\" (UID: \"928fb574-d543-486a-9d5b-ce3d48609cbb\") " pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:21.690795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.690715    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:21.962486 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:21.962406    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2jjsb"]
Sep 10 21:07:21.965652 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:07:21.965625    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928fb574_d543_486a_9d5b_ce3d48609cbb.slice/crio-56e938820ad42796e4041ce2a94cf4b15ef4147afdc6861dce8dc70bdf8c4ad9 WatchSource:0}: Error finding container 56e938820ad42796e4041ce2a94cf4b15ef4147afdc6861dce8dc70bdf8c4ad9: Status 404 returned error can't find the container with id 56e938820ad42796e4041ce2a94cf4b15ef4147afdc6861dce8dc70bdf8c4ad9
Sep 10 21:07:22.320348 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.320261    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-l4rjd"]
Sep 10 21:07:22.323566 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.323545    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l4rjd"
Sep 10 21:07:22.325725 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.325702    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-29nr7\""
Sep 10 21:07:22.331880 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.331858    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l4rjd"]
Sep 10 21:07:22.409306 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.409275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kmqt\" (UniqueName: \"kubernetes.io/projected/1fb407c1-9a08-433a-bc57-248b24498e0c-kube-api-access-8kmqt\") pod \"authorino-59c974bd48-l4rjd\" (UID: \"1fb407c1-9a08-433a-bc57-248b24498e0c\") " pod="kuadrant-system/authorino-59c974bd48-l4rjd"
Sep 10 21:07:22.510090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.510057    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kmqt\" (UniqueName: \"kubernetes.io/projected/1fb407c1-9a08-433a-bc57-248b24498e0c-kube-api-access-8kmqt\") pod \"authorino-59c974bd48-l4rjd\" (UID: \"1fb407c1-9a08-433a-bc57-248b24498e0c\") " pod="kuadrant-system/authorino-59c974bd48-l4rjd"
Sep 10 21:07:22.518961 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.518932    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kmqt\" (UniqueName: \"kubernetes.io/projected/1fb407c1-9a08-433a-bc57-248b24498e0c-kube-api-access-8kmqt\") pod \"authorino-59c974bd48-l4rjd\" (UID: \"1fb407c1-9a08-433a-bc57-248b24498e0c\") " pod="kuadrant-system/authorino-59c974bd48-l4rjd"
Sep 10 21:07:22.633494 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.633466    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l4rjd"
Sep 10 21:07:22.901141 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.901040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb" event={"ID":"928fb574-d543-486a-9d5b-ce3d48609cbb","Type":"ContainerStarted","Data":"56e938820ad42796e4041ce2a94cf4b15ef4147afdc6861dce8dc70bdf8c4ad9"}
Sep 10 21:07:22.907325 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:22.907301    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l4rjd"]
Sep 10 21:07:22.980515 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:07:22.980486    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1fb407c1_9a08_433a_bc57_248b24498e0c.slice/crio-7a8d9f3dada18b9af6cf2466b4b78be1676b70f64a858153eccdffc1a40646f1 WatchSource:0}: Error finding container 7a8d9f3dada18b9af6cf2466b4b78be1676b70f64a858153eccdffc1a40646f1: Status 404 returned error can't find the container with id 7a8d9f3dada18b9af6cf2466b4b78be1676b70f64a858153eccdffc1a40646f1
Sep 10 21:07:23.907170 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:23.907133    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l4rjd" event={"ID":"1fb407c1-9a08-433a-bc57-248b24498e0c","Type":"ContainerStarted","Data":"7a8d9f3dada18b9af6cf2466b4b78be1676b70f64a858153eccdffc1a40646f1"}
Sep 10 21:07:23.908680 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:23.908652    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb" event={"ID":"928fb574-d543-486a-9d5b-ce3d48609cbb","Type":"ContainerStarted","Data":"d481c5e2a42b137566c4e64fce3fe1eb28f20480d3c680b4e90a32a8e293d7c9"}
Sep 10 21:07:23.908838 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:23.908821    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:23.925793 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:23.925747    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb" podStartSLOduration=1.878662694 podStartE2EDuration="2.925732182s" totalImagesPullingTime="1.047069488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:23.924927117 +0000 UTC m=+530.927173477" watchObservedRunningTime="2026-09-10 21:07:23.925732182 +0000 UTC m=+530.927978559"
Sep 10 21:07:24.915057 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:24.915016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l4rjd" event={"ID":"1fb407c1-9a08-433a-bc57-248b24498e0c","Type":"ContainerStarted","Data":"8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c"}
Sep 10 21:07:24.929955 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:24.929911    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-l4rjd" podStartSLOduration=1.579943935 podStartE2EDuration="2.929896701s" totalImagesPullingTime="1.349952766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:24.928574238 +0000 UTC m=+531.930820588" watchObservedRunningTime="2026-09-10 21:07:24.929896701 +0000 UTC m=+531.932143047"
Sep 10 21:07:34.915770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:34.915738    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:37.456233 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:37.456197    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2jjsb"]
Sep 10 21:07:37.456611 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:37.456380    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb" podUID="928fb574-d543-486a-9d5b-ce3d48609cbb" containerName="limitador" containerID="cri-o://d481c5e2a42b137566c4e64fce3fe1eb28f20480d3c680b4e90a32a8e293d7c9" gracePeriod=30
Sep 10 21:07:37.985358 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:37.985325    2580 generic.go:356] "Generic (PLEG): container finished" podID="928fb574-d543-486a-9d5b-ce3d48609cbb" containerID="d481c5e2a42b137566c4e64fce3fe1eb28f20480d3c680b4e90a32a8e293d7c9" exitCode=0
Sep 10 21:07:37.985511 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:37.985402    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb" event={"ID":"928fb574-d543-486a-9d5b-ce3d48609cbb","Type":"ContainerDied","Data":"d481c5e2a42b137566c4e64fce3fe1eb28f20480d3c680b4e90a32a8e293d7c9"}
Sep 10 21:07:38.399229 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.399208    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:38.432367 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.432337    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/928fb574-d543-486a-9d5b-ce3d48609cbb-config-file\") pod \"928fb574-d543-486a-9d5b-ce3d48609cbb\" (UID: \"928fb574-d543-486a-9d5b-ce3d48609cbb\") "
Sep 10 21:07:38.432478 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.432453    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pdxl\" (UniqueName: \"kubernetes.io/projected/928fb574-d543-486a-9d5b-ce3d48609cbb-kube-api-access-5pdxl\") pod \"928fb574-d543-486a-9d5b-ce3d48609cbb\" (UID: \"928fb574-d543-486a-9d5b-ce3d48609cbb\") "
Sep 10 21:07:38.432660 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.432640    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928fb574-d543-486a-9d5b-ce3d48609cbb-config-file" (OuterVolumeSpecName: "config-file") pod "928fb574-d543-486a-9d5b-ce3d48609cbb" (UID: "928fb574-d543-486a-9d5b-ce3d48609cbb"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:07:38.434614 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.434583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928fb574-d543-486a-9d5b-ce3d48609cbb-kube-api-access-5pdxl" (OuterVolumeSpecName: "kube-api-access-5pdxl") pod "928fb574-d543-486a-9d5b-ce3d48609cbb" (UID: "928fb574-d543-486a-9d5b-ce3d48609cbb"). InnerVolumeSpecName "kube-api-access-5pdxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:07:38.534079 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.534000    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pdxl\" (UniqueName: \"kubernetes.io/projected/928fb574-d543-486a-9d5b-ce3d48609cbb-kube-api-access-5pdxl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:07:38.534079 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.534029    2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/928fb574-d543-486a-9d5b-ce3d48609cbb-config-file\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:07:38.992040 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.992003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb" event={"ID":"928fb574-d543-486a-9d5b-ce3d48609cbb","Type":"ContainerDied","Data":"56e938820ad42796e4041ce2a94cf4b15ef4147afdc6861dce8dc70bdf8c4ad9"}
Sep 10 21:07:38.992040 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.992027    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-2jjsb"
Sep 10 21:07:38.992300 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:38.992076    2580 scope.go:117] "RemoveContainer" containerID="d481c5e2a42b137566c4e64fce3fe1eb28f20480d3c680b4e90a32a8e293d7c9"
Sep 10 21:07:39.013406 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:39.013384    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2jjsb"]
Sep 10 21:07:39.017481 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:39.017462    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-2jjsb"]
Sep 10 21:07:39.523084 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:39.523046    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="928fb574-d543-486a-9d5b-ce3d48609cbb" path="/var/lib/kubelet/pods/928fb574-d543-486a-9d5b-ce3d48609cbb/volumes"
Sep 10 21:07:44.078554 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.078523    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"]
Sep 10 21:07:44.078906 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.078840    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928fb574-d543-486a-9d5b-ce3d48609cbb" containerName="limitador"
Sep 10 21:07:44.078906 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.078851    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="928fb574-d543-486a-9d5b-ce3d48609cbb" containerName="limitador"
Sep 10 21:07:44.078980 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.078925    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="928fb574-d543-486a-9d5b-ce3d48609cbb" containerName="limitador"
Sep 10 21:07:44.081933 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.081905    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:44.084183 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.084162    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 21:07:44.084411 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.084396    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-78fq7\""
Sep 10 21:07:44.084816 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.084797    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 21:07:44.084932 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.084871    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 21:07:44.090350 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.090330    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"]
Sep 10 21:07:44.176070 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.176037    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvbk\" (UniqueName: \"kubernetes.io/projected/f2b80b23-09d4-42fb-91ad-d710c658485a-kube-api-access-2jvbk\") pod \"postgres-69fb4594cf-pbtt2\" (UID: \"f2b80b23-09d4-42fb-91ad-d710c658485a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:44.176215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.176090    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f2b80b23-09d4-42fb-91ad-d710c658485a-data\") pod \"postgres-69fb4594cf-pbtt2\" (UID: \"f2b80b23-09d4-42fb-91ad-d710c658485a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:44.276828 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.276795    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvbk\" (UniqueName: \"kubernetes.io/projected/f2b80b23-09d4-42fb-91ad-d710c658485a-kube-api-access-2jvbk\") pod \"postgres-69fb4594cf-pbtt2\" (UID: \"f2b80b23-09d4-42fb-91ad-d710c658485a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:44.276960 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.276838    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f2b80b23-09d4-42fb-91ad-d710c658485a-data\") pod \"postgres-69fb4594cf-pbtt2\" (UID: \"f2b80b23-09d4-42fb-91ad-d710c658485a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:44.277194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.277179    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f2b80b23-09d4-42fb-91ad-d710c658485a-data\") pod \"postgres-69fb4594cf-pbtt2\" (UID: \"f2b80b23-09d4-42fb-91ad-d710c658485a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:44.285995 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.285970    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvbk\" (UniqueName: \"kubernetes.io/projected/f2b80b23-09d4-42fb-91ad-d710c658485a-kube-api-access-2jvbk\") pod \"postgres-69fb4594cf-pbtt2\" (UID: \"f2b80b23-09d4-42fb-91ad-d710c658485a\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:44.394186 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.394162    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:44.670421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:44.670392    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"]
Sep 10 21:07:44.671150 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:07:44.671100    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2b80b23_09d4_42fb_91ad_d710c658485a.slice/crio-fed54c101ca58dddb9763841308669e493161a77d23909e217a77320471da321 WatchSource:0}: Error finding container fed54c101ca58dddb9763841308669e493161a77d23909e217a77320471da321: Status 404 returned error can't find the container with id fed54c101ca58dddb9763841308669e493161a77d23909e217a77320471da321
Sep 10 21:07:45.022648 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:45.022565    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2" event={"ID":"f2b80b23-09d4-42fb-91ad-d710c658485a","Type":"ContainerStarted","Data":"fed54c101ca58dddb9763841308669e493161a77d23909e217a77320471da321"}
Sep 10 21:07:50.054937 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:50.054895    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2" event={"ID":"f2b80b23-09d4-42fb-91ad-d710c658485a","Type":"ContainerStarted","Data":"1ad8f16fa06d77e0acde9299142fa667e89c8e921af6b4fe81f7d803d722c84d"}
Sep 10 21:07:50.055328 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:50.054961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:50.079113 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:50.079064    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2" podStartSLOduration=1.282656617 podStartE2EDuration="6.079049513s" totalImagesPullingTime="4.796392896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:07:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:07:50.076995958 +0000 UTC m=+557.079242318" watchObservedRunningTime="2026-09-10 21:07:50.079049513 +0000 UTC m=+557.081295861"
Sep 10 21:07:56.088788 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:56.088752    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbtt2"
Sep 10 21:07:56.833861 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:56.833828    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dl7gp"]
Sep 10 21:07:56.836874 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:56.836855    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:56.839396 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:56.839378    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 21:07:56.846800 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:56.846772    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dl7gp"]
Sep 10 21:07:56.921315 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:56.921283    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dl7gp"]
Sep 10 21:07:56.921528 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:07:56.921510    2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-ststt tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-ststt tls-cert]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-dl7gp" podUID="ef36c7f5-94eb-4313-8996-b839cad40493"
Sep 10 21:07:56.984504 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:56.984467    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ststt\" (UniqueName: \"kubernetes.io/projected/ef36c7f5-94eb-4313-8996-b839cad40493-kube-api-access-ststt\") pod \"authorino-6d4dfc649f-dl7gp\" (UID: \"ef36c7f5-94eb-4313-8996-b839cad40493\") " pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:56.984666 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:56.984597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef36c7f5-94eb-4313-8996-b839cad40493-tls-cert\") pod \"authorino-6d4dfc649f-dl7gp\" (UID: \"ef36c7f5-94eb-4313-8996-b839cad40493\") " pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:57.085096 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.085016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef36c7f5-94eb-4313-8996-b839cad40493-tls-cert\") pod \"authorino-6d4dfc649f-dl7gp\" (UID: \"ef36c7f5-94eb-4313-8996-b839cad40493\") " pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:57.085096 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.085055    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ststt\" (UniqueName: \"kubernetes.io/projected/ef36c7f5-94eb-4313-8996-b839cad40493-kube-api-access-ststt\") pod \"authorino-6d4dfc649f-dl7gp\" (UID: \"ef36c7f5-94eb-4313-8996-b839cad40493\") " pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:57.087892 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.087859    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef36c7f5-94eb-4313-8996-b839cad40493-tls-cert\") pod \"authorino-6d4dfc649f-dl7gp\" (UID: \"ef36c7f5-94eb-4313-8996-b839cad40493\") " pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:57.094713 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.091543    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:57.098025 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.098005    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ststt\" (UniqueName: \"kubernetes.io/projected/ef36c7f5-94eb-4313-8996-b839cad40493-kube-api-access-ststt\") pod \"authorino-6d4dfc649f-dl7gp\" (UID: \"ef36c7f5-94eb-4313-8996-b839cad40493\") " pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:57.108775 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.108755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:57.185655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.185626    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef36c7f5-94eb-4313-8996-b839cad40493-tls-cert\") pod \"ef36c7f5-94eb-4313-8996-b839cad40493\" (UID: \"ef36c7f5-94eb-4313-8996-b839cad40493\") "
Sep 10 21:07:57.185815 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.185673    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ststt\" (UniqueName: \"kubernetes.io/projected/ef36c7f5-94eb-4313-8996-b839cad40493-kube-api-access-ststt\") pod \"ef36c7f5-94eb-4313-8996-b839cad40493\" (UID: \"ef36c7f5-94eb-4313-8996-b839cad40493\") "
Sep 10 21:07:57.187811 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.187782    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef36c7f5-94eb-4313-8996-b839cad40493-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ef36c7f5-94eb-4313-8996-b839cad40493" (UID: "ef36c7f5-94eb-4313-8996-b839cad40493"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:07:57.187899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.187840    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef36c7f5-94eb-4313-8996-b839cad40493-kube-api-access-ststt" (OuterVolumeSpecName: "kube-api-access-ststt") pod "ef36c7f5-94eb-4313-8996-b839cad40493" (UID: "ef36c7f5-94eb-4313-8996-b839cad40493"). InnerVolumeSpecName "kube-api-access-ststt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:07:57.286377 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.286342    2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ef36c7f5-94eb-4313-8996-b839cad40493-tls-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:07:57.286377 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:57.286375    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ststt\" (UniqueName: \"kubernetes.io/projected/ef36c7f5-94eb-4313-8996-b839cad40493-kube-api-access-ststt\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:07:58.096231 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:58.096200    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-dl7gp"
Sep 10 21:07:58.125150 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:58.125105    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dl7gp"]
Sep 10 21:07:58.132267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:58.132239    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-dl7gp"]
Sep 10 21:07:59.175271 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:59.175244    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l4rjd"]
Sep 10 21:07:59.175644 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:59.175400    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-l4rjd" podUID="1fb407c1-9a08-433a-bc57-248b24498e0c" containerName="authorino" containerID="cri-o://8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c" gracePeriod=30
Sep 10 21:07:59.415203 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:59.415180    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l4rjd"
Sep 10 21:07:59.506032 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:59.505959    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kmqt\" (UniqueName: \"kubernetes.io/projected/1fb407c1-9a08-433a-bc57-248b24498e0c-kube-api-access-8kmqt\") pod \"1fb407c1-9a08-433a-bc57-248b24498e0c\" (UID: \"1fb407c1-9a08-433a-bc57-248b24498e0c\") "
Sep 10 21:07:59.508047 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:59.508022    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb407c1-9a08-433a-bc57-248b24498e0c-kube-api-access-8kmqt" (OuterVolumeSpecName: "kube-api-access-8kmqt") pod "1fb407c1-9a08-433a-bc57-248b24498e0c" (UID: "1fb407c1-9a08-433a-bc57-248b24498e0c"). InnerVolumeSpecName "kube-api-access-8kmqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:07:59.519013 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:59.518990    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef36c7f5-94eb-4313-8996-b839cad40493" path="/var/lib/kubelet/pods/ef36c7f5-94eb-4313-8996-b839cad40493/volumes"
Sep 10 21:07:59.606770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:07:59.606743    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kmqt\" (UniqueName: \"kubernetes.io/projected/1fb407c1-9a08-433a-bc57-248b24498e0c-kube-api-access-8kmqt\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:08:00.108990 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.108952    2580 generic.go:356] "Generic (PLEG): container finished" podID="1fb407c1-9a08-433a-bc57-248b24498e0c" containerID="8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c" exitCode=0
Sep 10 21:08:00.109211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.109025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-l4rjd"
Sep 10 21:08:00.109211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.109038    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l4rjd" event={"ID":"1fb407c1-9a08-433a-bc57-248b24498e0c","Type":"ContainerDied","Data":"8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c"}
Sep 10 21:08:00.109211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.109085    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-l4rjd" event={"ID":"1fb407c1-9a08-433a-bc57-248b24498e0c","Type":"ContainerDied","Data":"7a8d9f3dada18b9af6cf2466b4b78be1676b70f64a858153eccdffc1a40646f1"}
Sep 10 21:08:00.109211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.109107    2580 scope.go:117] "RemoveContainer" containerID="8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c"
Sep 10 21:08:00.118443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.118419    2580 scope.go:117] "RemoveContainer" containerID="8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c"
Sep 10 21:08:00.118676 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:08:00.118660    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c\": container with ID starting with 8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c not found: ID does not exist" containerID="8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c"
Sep 10 21:08:00.118732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.118682    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c"} err="failed to get container status \"8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c\": rpc error: code = NotFound desc = could not find container \"8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c\": container with ID starting with 8a9c071f814a4dabeffd937d6068a4e826e12775d654a3e3f86dd2852a0b875c not found: ID does not exist"
Sep 10 21:08:00.127892 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.127859    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l4rjd"]
Sep 10 21:08:00.132638 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:00.132614    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-l4rjd"]
Sep 10 21:08:01.518519 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:01.518492    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fb407c1-9a08-433a-bc57-248b24498e0c" path="/var/lib/kubelet/pods/1fb407c1-9a08-433a-bc57-248b24498e0c/volumes"
Sep 10 21:08:04.220091 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.220062    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-766fc89467-q982w"]
Sep 10 21:08:04.220474 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.220451    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fb407c1-9a08-433a-bc57-248b24498e0c" containerName="authorino"
Sep 10 21:08:04.220474 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.220466    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fb407c1-9a08-433a-bc57-248b24498e0c" containerName="authorino"
Sep 10 21:08:04.220552 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.220533    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fb407c1-9a08-433a-bc57-248b24498e0c" containerName="authorino"
Sep 10 21:08:04.224859 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.224829    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.230169 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.230145    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6h5kw\""
Sep 10 21:08:04.230305 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.230194    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 21:08:04.230305 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.230237    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 21:08:04.230612 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.230582    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 21:08:04.245291 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.245270    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766fc89467-q982w"]
Sep 10 21:08:04.344792 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.344762    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.344942 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.344805    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgrp8\" (UniqueName: \"kubernetes.io/projected/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-kube-api-access-bgrp8\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.344942 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.344875    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.406927 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.406888    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84497f567c-l4ddm"]
Sep 10 21:08:04.410071 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.410050    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.424713 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.424692    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84497f567c-l4ddm"]
Sep 10 21:08:04.446230 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.446207    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.446321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.446249    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgrp8\" (UniqueName: \"kubernetes.io/projected/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-kube-api-access-bgrp8\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.446321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.446296    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.446403 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:08:04.446359    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 21:08:04.446453 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:08:04.446403    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 10 21:08:04.446453 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:08:04.446419    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs podName:adc3bdab-3373-4d5e-9716-6a3e85ab16a3 nodeName:}" failed. No retries permitted until 2026-09-10 21:08:04.94640373 +0000 UTC m=+571.948650058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs") pod "maas-controller-766fc89467-q982w" (UID: "adc3bdab-3373-4d5e-9716-6a3e85ab16a3") : secret "maas-controller-metrics-tls" not found
Sep 10 21:08:04.446453 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:08:04.446450    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert podName:adc3bdab-3373-4d5e-9716-6a3e85ab16a3 nodeName:}" failed. No retries permitted until 2026-09-10 21:08:04.946438029 +0000 UTC m=+571.948684353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert") pod "maas-controller-766fc89467-q982w" (UID: "adc3bdab-3373-4d5e-9716-6a3e85ab16a3") : secret "maas-controller-webhook-cert" not found
Sep 10 21:08:04.458681 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.458658    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgrp8\" (UniqueName: \"kubernetes.io/projected/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-kube-api-access-bgrp8\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.547395 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.547320    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.547395 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.547354    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cjmk\" (UniqueName: \"kubernetes.io/projected/c65b4183-378d-4533-91a1-7ed2824994d7-kube-api-access-8cjmk\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.547540 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.547428    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-webhook-cert\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.648074 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.648042    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.648074 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.648074    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cjmk\" (UniqueName: \"kubernetes.io/projected/c65b4183-378d-4533-91a1-7ed2824994d7-kube-api-access-8cjmk\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.648329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.648092    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-webhook-cert\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.648329 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:08:04.648220    2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 21:08:04.648329 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:08:04.648293    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs podName:c65b4183-378d-4533-91a1-7ed2824994d7 nodeName:}" failed. No retries permitted until 2026-09-10 21:08:05.148273513 +0000 UTC m=+572.150519840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs") pod "maas-controller-84497f567c-l4ddm" (UID: "c65b4183-378d-4533-91a1-7ed2824994d7") : secret "maas-controller-metrics-tls" not found
Sep 10 21:08:04.650772 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.650752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-webhook-cert\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.659311 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.659278    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cjmk\" (UniqueName: \"kubernetes.io/projected/c65b4183-378d-4533-91a1-7ed2824994d7-kube-api-access-8cjmk\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:04.950422 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.950390    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.950590 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.950442    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.952983 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.952955    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:04.953153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:04.953115    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert\") pod \"maas-controller-766fc89467-q982w\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") " pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:05.134989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:05.134957    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:05.152343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:05.152300    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:05.154652 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:05.154625    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs\") pod \"maas-controller-84497f567c-l4ddm\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") " pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:05.319539 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:05.319459    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:05.407824 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:05.407800    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-766fc89467-q982w"]
Sep 10 21:08:05.409476 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:08:05.409449    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadc3bdab_3373_4d5e_9716_6a3e85ab16a3.slice/crio-a1b509c122441ba83dd0214644c05bafb3f17202a68b1acce7c7812a8e952de9 WatchSource:0}: Error finding container a1b509c122441ba83dd0214644c05bafb3f17202a68b1acce7c7812a8e952de9: Status 404 returned error can't find the container with id a1b509c122441ba83dd0214644c05bafb3f17202a68b1acce7c7812a8e952de9
Sep 10 21:08:05.590683 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:05.590654    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84497f567c-l4ddm"]
Sep 10 21:08:05.592184 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:08:05.592158    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65b4183_378d_4533_91a1_7ed2824994d7.slice/crio-6929bb72121227b7b429850f2fcec6bc9a664b6eefc9f333535de58e9b419e27 WatchSource:0}: Error finding container 6929bb72121227b7b429850f2fcec6bc9a664b6eefc9f333535de58e9b419e27: Status 404 returned error can't find the container with id 6929bb72121227b7b429850f2fcec6bc9a664b6eefc9f333535de58e9b419e27
Sep 10 21:08:06.141785 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:06.141749    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766fc89467-q982w" event={"ID":"adc3bdab-3373-4d5e-9716-6a3e85ab16a3","Type":"ContainerStarted","Data":"a1b509c122441ba83dd0214644c05bafb3f17202a68b1acce7c7812a8e952de9"}
Sep 10 21:08:06.142989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:06.142966    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-l4ddm" event={"ID":"c65b4183-378d-4533-91a1-7ed2824994d7","Type":"ContainerStarted","Data":"6929bb72121227b7b429850f2fcec6bc9a664b6eefc9f333535de58e9b419e27"}
Sep 10 21:08:09.160632 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:09.160598    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766fc89467-q982w" event={"ID":"adc3bdab-3373-4d5e-9716-6a3e85ab16a3","Type":"ContainerStarted","Data":"102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77"}
Sep 10 21:08:09.161089 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:09.160667    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:09.162184 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:09.162162    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-l4ddm" event={"ID":"c65b4183-378d-4533-91a1-7ed2824994d7","Type":"ContainerStarted","Data":"0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84"}
Sep 10 21:08:09.162286 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:09.162215    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:09.185037 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:09.184354    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-766fc89467-q982w" podStartSLOduration=2.3260143429999998 podStartE2EDuration="5.184335841s" totalImagesPullingTime="2.858321498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:09.183510176 +0000 UTC m=+576.185756524" watchObservedRunningTime="2026-09-10 21:08:09.184335841 +0000 UTC m=+576.186582188"
Sep 10 21:08:09.217257 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:09.217213    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84497f567c-l4ddm" podStartSLOduration=2.535380629 podStartE2EDuration="5.217196741s" totalImagesPullingTime="2.681816112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:09.215980848 +0000 UTC m=+576.218227196" watchObservedRunningTime="2026-09-10 21:08:09.217196741 +0000 UTC m=+576.219443088"
Sep 10 21:08:20.170451 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.170409    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:08:20.170836 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.170491    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:20.244995 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.244960    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766fc89467-q982w"]
Sep 10 21:08:20.245225 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.245198    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-766fc89467-q982w" podUID="adc3bdab-3373-4d5e-9716-6a3e85ab16a3" containerName="manager" containerID="cri-o://102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77" gracePeriod=30
Sep 10 21:08:20.493623 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.493601    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:20.586459 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.586432    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert\") pod \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") "
Sep 10 21:08:20.586632 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.586509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs\") pod \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") "
Sep 10 21:08:20.586692 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.586641    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgrp8\" (UniqueName: \"kubernetes.io/projected/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-kube-api-access-bgrp8\") pod \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\" (UID: \"adc3bdab-3373-4d5e-9716-6a3e85ab16a3\") "
Sep 10 21:08:20.588852 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.588817    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "adc3bdab-3373-4d5e-9716-6a3e85ab16a3" (UID: "adc3bdab-3373-4d5e-9716-6a3e85ab16a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:08:20.589020 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.588875    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "adc3bdab-3373-4d5e-9716-6a3e85ab16a3" (UID: "adc3bdab-3373-4d5e-9716-6a3e85ab16a3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:08:20.589020 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.588889    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-kube-api-access-bgrp8" (OuterVolumeSpecName: "kube-api-access-bgrp8") pod "adc3bdab-3373-4d5e-9716-6a3e85ab16a3" (UID: "adc3bdab-3373-4d5e-9716-6a3e85ab16a3"). InnerVolumeSpecName "kube-api-access-bgrp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:08:20.687297 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.687270    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgrp8\" (UniqueName: \"kubernetes.io/projected/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-kube-api-access-bgrp8\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:08:20.687297 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.687295    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-webhook-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:08:20.687522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:20.687310    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/adc3bdab-3373-4d5e-9716-6a3e85ab16a3-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:08:21.230096 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.230061    2580 generic.go:356] "Generic (PLEG): container finished" podID="adc3bdab-3373-4d5e-9716-6a3e85ab16a3" containerID="102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77" exitCode=0
Sep 10 21:08:21.230565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.230106    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-766fc89467-q982w"
Sep 10 21:08:21.230565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.230153    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766fc89467-q982w" event={"ID":"adc3bdab-3373-4d5e-9716-6a3e85ab16a3","Type":"ContainerDied","Data":"102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77"}
Sep 10 21:08:21.230565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.230223    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-766fc89467-q982w" event={"ID":"adc3bdab-3373-4d5e-9716-6a3e85ab16a3","Type":"ContainerDied","Data":"a1b509c122441ba83dd0214644c05bafb3f17202a68b1acce7c7812a8e952de9"}
Sep 10 21:08:21.230565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.230258    2580 scope.go:117] "RemoveContainer" containerID="102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77"
Sep 10 21:08:21.239561 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.239545    2580 scope.go:117] "RemoveContainer" containerID="102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77"
Sep 10 21:08:21.239807 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:08:21.239787    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77\": container with ID starting with 102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77 not found: ID does not exist" containerID="102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77"
Sep 10 21:08:21.239858 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.239815    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77"} err="failed to get container status \"102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77\": rpc error: code = NotFound desc = could not find container \"102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77\": container with ID starting with 102d24c49b4171492e46561878703b2dbe66973e9bd49b48170b64f5c5a92a77 not found: ID does not exist"
Sep 10 21:08:21.253638 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.253619    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-766fc89467-q982w"]
Sep 10 21:08:21.259601 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.259582    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-766fc89467-q982w"]
Sep 10 21:08:21.518923 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:21.518853    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adc3bdab-3373-4d5e-9716-6a3e85ab16a3" path="/var/lib/kubelet/pods/adc3bdab-3373-4d5e-9716-6a3e85ab16a3/volumes"
Sep 10 21:08:42.828001 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.827968    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-nxd8f"]
Sep 10 21:08:42.833328 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.828309    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adc3bdab-3373-4d5e-9716-6a3e85ab16a3" containerName="manager"
Sep 10 21:08:42.833328 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.828320    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="adc3bdab-3373-4d5e-9716-6a3e85ab16a3" containerName="manager"
Sep 10 21:08:42.833328 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.828373    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="adc3bdab-3373-4d5e-9716-6a3e85ab16a3" containerName="manager"
Sep 10 21:08:42.833609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.833594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.836687 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.836657    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 21:08:42.836687 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.836658    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 21:08:42.837099 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.836699    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-njbnl\""
Sep 10 21:08:42.847889 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.847868    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-nxd8f"]
Sep 10 21:08:42.866938 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.866910    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-metrics-certs\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.867065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.866992    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-maas-api-tls\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.867113 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.867074    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02867d1d-96c4-493a-95f9-b9a3711b0808-openshift-service-ca\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.867188 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.867107    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqtk4\" (UniqueName: \"kubernetes.io/projected/02867d1d-96c4-493a-95f9-b9a3711b0808-kube-api-access-rqtk4\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.967528 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.967496    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-metrics-certs\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.967528 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.967536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-maas-api-tls\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.967780 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.967571    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02867d1d-96c4-493a-95f9-b9a3711b0808-openshift-service-ca\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.967780 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.967587    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqtk4\" (UniqueName: \"kubernetes.io/projected/02867d1d-96c4-493a-95f9-b9a3711b0808-kube-api-access-rqtk4\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.968237 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.968218    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02867d1d-96c4-493a-95f9-b9a3711b0808-openshift-service-ca\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.970041 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.970022    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-maas-api-tls\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.970158 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.970082    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-metrics-certs\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:42.979371 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:42.979346    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqtk4\" (UniqueName: \"kubernetes.io/projected/02867d1d-96c4-493a-95f9-b9a3711b0808-kube-api-access-rqtk4\") pod \"maas-api-554988677c-nxd8f\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") " pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:43.143704 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.143671    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:43.425395 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.425369    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-nxd8f"]
Sep 10 21:08:43.426695 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:08:43.426661    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02867d1d_96c4_493a_95f9_b9a3711b0808.slice/crio-9639896b1b5abca2f68d6cfd06b5795ddc6030848a758c8e7e2d0a060fe27f7b WatchSource:0}: Error finding container 9639896b1b5abca2f68d6cfd06b5795ddc6030848a758c8e7e2d0a060fe27f7b: Status 404 returned error can't find the container with id 9639896b1b5abca2f68d6cfd06b5795ddc6030848a758c8e7e2d0a060fe27f7b
Sep 10 21:08:43.427992 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.427975    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:08:43.452692 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.452665    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mg8df"]
Sep 10 21:08:43.457189 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.457168    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:43.460465 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.460441    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6pvvs\""
Sep 10 21:08:43.460561 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.460440    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 21:08:43.466773 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.466749    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mg8df"]
Sep 10 21:08:43.573002 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.572974    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxnkd\" (UniqueName: \"kubernetes.io/projected/1aa96712-b017-464c-b322-0ef8cbd99c8f-kube-api-access-lxnkd\") pod \"payload-processing-5b54b7fbb7-mg8df\" (UID: \"1aa96712-b017-464c-b322-0ef8cbd99c8f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:43.573231 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.573199    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa96712-b017-464c-b322-0ef8cbd99c8f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mg8df\" (UID: \"1aa96712-b017-464c-b322-0ef8cbd99c8f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:43.674442 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.674351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa96712-b017-464c-b322-0ef8cbd99c8f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mg8df\" (UID: \"1aa96712-b017-464c-b322-0ef8cbd99c8f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:43.674584 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.674445    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxnkd\" (UniqueName: \"kubernetes.io/projected/1aa96712-b017-464c-b322-0ef8cbd99c8f-kube-api-access-lxnkd\") pod \"payload-processing-5b54b7fbb7-mg8df\" (UID: \"1aa96712-b017-464c-b322-0ef8cbd99c8f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:43.675010 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.674993    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1aa96712-b017-464c-b322-0ef8cbd99c8f-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-mg8df\" (UID: \"1aa96712-b017-464c-b322-0ef8cbd99c8f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:43.686805 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.686775    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxnkd\" (UniqueName: \"kubernetes.io/projected/1aa96712-b017-464c-b322-0ef8cbd99c8f-kube-api-access-lxnkd\") pod \"payload-processing-5b54b7fbb7-mg8df\" (UID: \"1aa96712-b017-464c-b322-0ef8cbd99c8f\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:43.769925 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:43.769889    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:44.054039 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:44.048611    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-mg8df"]
Sep 10 21:08:44.349619 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:44.349527    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f" event={"ID":"02867d1d-96c4-493a-95f9-b9a3711b0808","Type":"ContainerStarted","Data":"9639896b1b5abca2f68d6cfd06b5795ddc6030848a758c8e7e2d0a060fe27f7b"}
Sep 10 21:08:44.350981 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:44.350940    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df" event={"ID":"1aa96712-b017-464c-b322-0ef8cbd99c8f","Type":"ContainerStarted","Data":"42393384980b3d0fd68a42704eea3be57e2558b86c0fc3d6b64a16639d4f485b"}
Sep 10 21:08:45.358739 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:45.358704    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f" event={"ID":"02867d1d-96c4-493a-95f9-b9a3711b0808","Type":"ContainerStarted","Data":"48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f"}
Sep 10 21:08:45.359210 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:45.358801    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:45.385191 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:45.385139    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f" podStartSLOduration=1.580275877 podStartE2EDuration="3.385105623s" totalImagesPullingTime="1.804829746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:45.38319247 +0000 UTC m=+612.385438819" watchObservedRunningTime="2026-09-10 21:08:45.385105623 +0000 UTC m=+612.387351969"
Sep 10 21:08:46.366659 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:46.366619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df" event={"ID":"1aa96712-b017-464c-b322-0ef8cbd99c8f","Type":"ContainerStarted","Data":"603cf2c885512ec6da4d5a180059eeaffa70d2729e85099eb76c214874e8efce"}
Sep 10 21:08:46.367156 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:46.366841    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:08:46.385093 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:46.385050    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df" podStartSLOduration=1.6873917889999999 podStartE2EDuration="3.385028915s" totalImagesPullingTime="1.697637126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:08:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:08:46.384174037 +0000 UTC m=+613.386420384" watchObservedRunningTime="2026-09-10 21:08:46.385028915 +0000 UTC m=+613.387275261"
Sep 10 21:08:51.371921 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:51.371887    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:08:57.373172 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:08:57.373139    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-mg8df"
Sep 10 21:09:18.576721 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.576690    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"]
Sep 10 21:09:18.581136 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.581099    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.583844 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.583822    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 21:09:18.584213 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.583826    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kxt89\""
Sep 10 21:09:18.584496 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.584464    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 21:09:18.584596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.584469    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 21:09:18.595315 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.595295    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"]
Sep 10 21:09:18.678475 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.678438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.678646 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.678483    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.678646 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.678577    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013c0199-7a19-460d-8352-3d3cc3ed4062-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.678646 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.678634    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.678819 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.678677    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr5hq\" (UniqueName: \"kubernetes.io/projected/013c0199-7a19-460d-8352-3d3cc3ed4062-kube-api-access-dr5hq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.678819 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.678699    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.779751 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.779720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.779890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.779757    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.779890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.779785    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013c0199-7a19-460d-8352-3d3cc3ed4062-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.779890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.779823    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.779890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.779848    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr5hq\" (UniqueName: \"kubernetes.io/projected/013c0199-7a19-460d-8352-3d3cc3ed4062-kube-api-access-dr5hq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.779890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.779864    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.780226 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.780191    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.780285 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.780234    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.780320 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.780300    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.782299 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.782276    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/013c0199-7a19-460d-8352-3d3cc3ed4062-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.782454 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.782431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/013c0199-7a19-460d-8352-3d3cc3ed4062-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.789379 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.789359    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr5hq\" (UniqueName: \"kubernetes.io/projected/013c0199-7a19-460d-8352-3d3cc3ed4062-kube-api-access-dr5hq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vswwq\" (UID: \"013c0199-7a19-460d-8352-3d3cc3ed4062\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:18.892291 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:18.892265    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:19.174910 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:19.174877    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"]
Sep 10 21:09:19.176751 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:09:19.176720    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod013c0199_7a19_460d_8352_3d3cc3ed4062.slice/crio-6fc3a8cb981eba9008c025aa5f2ac42722d5895f1a30953c8c1e92a18ca6e096 WatchSource:0}: Error finding container 6fc3a8cb981eba9008c025aa5f2ac42722d5895f1a30953c8c1e92a18ca6e096: Status 404 returned error can't find the container with id 6fc3a8cb981eba9008c025aa5f2ac42722d5895f1a30953c8c1e92a18ca6e096
Sep 10 21:09:19.540947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:19.540865    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq" event={"ID":"013c0199-7a19-460d-8352-3d3cc3ed4062","Type":"ContainerStarted","Data":"6fc3a8cb981eba9008c025aa5f2ac42722d5895f1a30953c8c1e92a18ca6e096"}
Sep 10 21:09:20.547664 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:20.547587    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq" event={"ID":"013c0199-7a19-460d-8352-3d3cc3ed4062","Type":"ContainerStarted","Data":"a9a68d3f192cf68fc3170936215e078d1ed4a67beec74ea3d4b4431e2743a0de"}
Sep 10 21:09:20.567335 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:20.567287    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq" podStartSLOduration=1.594943346 podStartE2EDuration="2.567271829s" totalImagesPullingTime="972.328483ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:09:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:09:20.565512968 +0000 UTC m=+647.567759315" watchObservedRunningTime="2026-09-10 21:09:20.567271829 +0000 UTC m=+647.569518176"
Sep 10 21:09:28.892927 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:28.892889    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:28.893428 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:28.892954    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:28.905718 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:28.905694    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:09:29.602243 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:09:29.602218    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vswwq"
Sep 10 21:11:30.091707 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.091669    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57c46d699b-thm79"]
Sep 10 21:11:30.095114 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.095087    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c46d699b-thm79"
Sep 10 21:11:30.097991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.097969    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-29nr7\""
Sep 10 21:11:30.098108 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.097981    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 21:11:30.102884 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.102471    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c46d699b-thm79"]
Sep 10 21:11:30.142269 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.142241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ndm\" (UniqueName: \"kubernetes.io/projected/f227f4ee-7aa6-4b25-a136-ef6e86804794-kube-api-access-m7ndm\") pod \"authorino-57c46d699b-thm79\" (UID: \"f227f4ee-7aa6-4b25-a136-ef6e86804794\") " pod="kuadrant-system/authorino-57c46d699b-thm79"
Sep 10 21:11:30.142434 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.142293    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f227f4ee-7aa6-4b25-a136-ef6e86804794-tls-cert\") pod \"authorino-57c46d699b-thm79\" (UID: \"f227f4ee-7aa6-4b25-a136-ef6e86804794\") " pod="kuadrant-system/authorino-57c46d699b-thm79"
Sep 10 21:11:30.243574 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.243543    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f227f4ee-7aa6-4b25-a136-ef6e86804794-tls-cert\") pod \"authorino-57c46d699b-thm79\" (UID: \"f227f4ee-7aa6-4b25-a136-ef6e86804794\") " pod="kuadrant-system/authorino-57c46d699b-thm79"
Sep 10 21:11:30.243765 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.243612    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ndm\" (UniqueName: \"kubernetes.io/projected/f227f4ee-7aa6-4b25-a136-ef6e86804794-kube-api-access-m7ndm\") pod \"authorino-57c46d699b-thm79\" (UID: \"f227f4ee-7aa6-4b25-a136-ef6e86804794\") " pod="kuadrant-system/authorino-57c46d699b-thm79"
Sep 10 21:11:30.246097 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.246072    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/f227f4ee-7aa6-4b25-a136-ef6e86804794-tls-cert\") pod \"authorino-57c46d699b-thm79\" (UID: \"f227f4ee-7aa6-4b25-a136-ef6e86804794\") " pod="kuadrant-system/authorino-57c46d699b-thm79"
Sep 10 21:11:30.251378 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.251353    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ndm\" (UniqueName: \"kubernetes.io/projected/f227f4ee-7aa6-4b25-a136-ef6e86804794-kube-api-access-m7ndm\") pod \"authorino-57c46d699b-thm79\" (UID: \"f227f4ee-7aa6-4b25-a136-ef6e86804794\") " pod="kuadrant-system/authorino-57c46d699b-thm79"
Sep 10 21:11:30.406005 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.405978    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57c46d699b-thm79"
Sep 10 21:11:30.677837 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:30.677807    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57c46d699b-thm79"]
Sep 10 21:11:30.679210 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:11:30.679186    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf227f4ee_7aa6_4b25_a136_ef6e86804794.slice/crio-040fc791f7fe075b75c83dc26e409f8b781cf8b382d2fd6b386c8038cd561143 WatchSource:0}: Error finding container 040fc791f7fe075b75c83dc26e409f8b781cf8b382d2fd6b386c8038cd561143: Status 404 returned error can't find the container with id 040fc791f7fe075b75c83dc26e409f8b781cf8b382d2fd6b386c8038cd561143
Sep 10 21:11:31.227963 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:31.227924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c46d699b-thm79" event={"ID":"f227f4ee-7aa6-4b25-a136-ef6e86804794","Type":"ContainerStarted","Data":"040fc791f7fe075b75c83dc26e409f8b781cf8b382d2fd6b386c8038cd561143"}
Sep 10 21:11:32.235658 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:32.235619    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57c46d699b-thm79" event={"ID":"f227f4ee-7aa6-4b25-a136-ef6e86804794","Type":"ContainerStarted","Data":"eda0f174dfc0e9be995686e69afc5d327c86296ec7c58dc3bf5e096ce6ec1ba7"}
Sep 10 21:11:32.254845 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:32.254792    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57c46d699b-thm79" podStartSLOduration=1.5335977939999998 podStartE2EDuration="2.254775313s" totalImagesPullingTime="721.177519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:32.25447362 +0000 UTC m=+779.256719971" watchObservedRunningTime="2026-09-10 21:11:32.254775313 +0000 UTC m=+779.257021661"
Sep 10 21:11:52.031009 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.030979    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8134-efc1e0"]
Sep 10 21:11:52.042024 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.041996    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8134-efc1e0"]
Sep 10 21:11:52.042226 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.042093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:11:52.044260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.044236    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-cc2nx\""
Sep 10 21:11:52.145289 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.145259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrv8f\" (UniqueName: \"kubernetes.io/projected/e719ed0d-97d8-49ac-b2da-c5200b349501-kube-api-access-xrv8f\") pod \"test-curl-8134-efc1e0\" (UID: \"e719ed0d-97d8-49ac-b2da-c5200b349501\") " pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:11:52.245942 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.245911    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrv8f\" (UniqueName: \"kubernetes.io/projected/e719ed0d-97d8-49ac-b2da-c5200b349501-kube-api-access-xrv8f\") pod \"test-curl-8134-efc1e0\" (UID: \"e719ed0d-97d8-49ac-b2da-c5200b349501\") " pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:11:52.256046 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.256018    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrv8f\" (UniqueName: \"kubernetes.io/projected/e719ed0d-97d8-49ac-b2da-c5200b349501-kube-api-access-xrv8f\") pod \"test-curl-8134-efc1e0\" (UID: \"e719ed0d-97d8-49ac-b2da-c5200b349501\") " pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:11:52.351767 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.351689    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:11:52.621547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:52.621517    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8134-efc1e0"]
Sep 10 21:11:52.624408 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:11:52.624381    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode719ed0d_97d8_49ac_b2da_c5200b349501.slice/crio-8b6ad8b21cb8c54029716a8bbf5134039c5009d1a4d5d1d58c45847987d3285a WatchSource:0}: Error finding container 8b6ad8b21cb8c54029716a8bbf5134039c5009d1a4d5d1d58c45847987d3285a: Status 404 returned error can't find the container with id 8b6ad8b21cb8c54029716a8bbf5134039c5009d1a4d5d1d58c45847987d3285a
Sep 10 21:11:53.349632 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:53.349599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8134-efc1e0" event={"ID":"e719ed0d-97d8-49ac-b2da-c5200b349501","Type":"ContainerStarted","Data":"8b6ad8b21cb8c54029716a8bbf5134039c5009d1a4d5d1d58c45847987d3285a"}
Sep 10 21:11:54.341853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.341820    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"]
Sep 10 21:11:54.361238 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.361210    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"]
Sep 10 21:11:54.361702 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.361377    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.363564 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.363540    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-b22a78-serving-cert\""
Sep 10 21:11:54.466304 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.466259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvfzb\" (UniqueName: \"kubernetes.io/projected/c577d9d4-9bee-47f3-82bd-1d5a68caba13-kube-api-access-pvfzb\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.466514 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.466337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-metrics-certs\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.466514 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.466377    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c577d9d4-9bee-47f3-82bd-1d5a68caba13-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.466514 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.466460    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-maas-api-tls\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.567916 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.567833    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-maas-api-tls\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.568083 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.567925    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvfzb\" (UniqueName: \"kubernetes.io/projected/c577d9d4-9bee-47f3-82bd-1d5a68caba13-kube-api-access-pvfzb\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.568083 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.567975    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-metrics-certs\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.568274 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.568097    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c577d9d4-9bee-47f3-82bd-1d5a68caba13-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.568776 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.568742    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c577d9d4-9bee-47f3-82bd-1d5a68caba13-openshift-service-ca\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.570873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.570830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-maas-api-tls\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.570995 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.570930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-metrics-certs\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.580548 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.580489    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvfzb\" (UniqueName: \"kubernetes.io/projected/c577d9d4-9bee-47f3-82bd-1d5a68caba13-kube-api-access-pvfzb\") pod \"maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.675157 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.675127    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:54.987167 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:54.987140    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"]
Sep 10 21:11:54.990513 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:11:54.990480    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc577d9d4_9bee_47f3_82bd_1d5a68caba13.slice/crio-91d23a66c5efd79281bd7c0c33f97d29518af8a50f12d327887df604bbe241c4 WatchSource:0}: Error finding container 91d23a66c5efd79281bd7c0c33f97d29518af8a50f12d327887df604bbe241c4: Status 404 returned error can't find the container with id 91d23a66c5efd79281bd7c0c33f97d29518af8a50f12d327887df604bbe241c4
Sep 10 21:11:55.363760 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:55.363678    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8134-efc1e0" event={"ID":"e719ed0d-97d8-49ac-b2da-c5200b349501","Type":"ContainerStarted","Data":"f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42"}
Sep 10 21:11:55.365052 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:55.365026    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb" event={"ID":"c577d9d4-9bee-47f3-82bd-1d5a68caba13","Type":"ContainerStarted","Data":"91d23a66c5efd79281bd7c0c33f97d29518af8a50f12d327887df604bbe241c4"}
Sep 10 21:11:55.378654 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:55.378604    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8134-efc1e0" podStartSLOduration=1.221701369 podStartE2EDuration="3.378585856s" totalImagesPullingTime="2.156884487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:55.377823749 +0000 UTC m=+802.380070097" watchObservedRunningTime="2026-09-10 21:11:55.378585856 +0000 UTC m=+802.380832206"
Sep 10 21:11:55.987208 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:55.987164    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8134-efc1e0"]
Sep 10 21:11:55.992174 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:55.992144    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8134-efc1e0"]
Sep 10 21:11:56.117281 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.117210    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8134-238617"]
Sep 10 21:11:56.117588 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.117576    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e719ed0d-97d8-49ac-b2da-c5200b349501" containerName="test-curl-8134-efc1e0"
Sep 10 21:11:56.117631 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.117590    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e719ed0d-97d8-49ac-b2da-c5200b349501" containerName="test-curl-8134-efc1e0"
Sep 10 21:11:56.117673 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.117665    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e719ed0d-97d8-49ac-b2da-c5200b349501" containerName="test-curl-8134-efc1e0"
Sep 10 21:11:56.120483 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.120461    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:11:56.125619 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.125588    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8134-238617"]
Sep 10 21:11:56.182433 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.182390    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrw6\" (UniqueName: \"kubernetes.io/projected/2dd08870-d48e-4137-bb3d-b1bd3dc75308-kube-api-access-ncrw6\") pod \"test-curl-8134-238617\" (UID: \"2dd08870-d48e-4137-bb3d-b1bd3dc75308\") " pod="opendatahub/test-curl-8134-238617"
Sep 10 21:11:56.283226 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.283197    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrw6\" (UniqueName: \"kubernetes.io/projected/2dd08870-d48e-4137-bb3d-b1bd3dc75308-kube-api-access-ncrw6\") pod \"test-curl-8134-238617\" (UID: \"2dd08870-d48e-4137-bb3d-b1bd3dc75308\") " pod="opendatahub/test-curl-8134-238617"
Sep 10 21:11:56.291396 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.291368    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrw6\" (UniqueName: \"kubernetes.io/projected/2dd08870-d48e-4137-bb3d-b1bd3dc75308-kube-api-access-ncrw6\") pod \"test-curl-8134-238617\" (UID: \"2dd08870-d48e-4137-bb3d-b1bd3dc75308\") " pod="opendatahub/test-curl-8134-238617"
Sep 10 21:11:56.372854 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.372774    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb" event={"ID":"c577d9d4-9bee-47f3-82bd-1d5a68caba13","Type":"ContainerStarted","Data":"9904900eeed284f6dd395ae45b49614c51f46d2a8249cf4f691d68e52096bdef"}
Sep 10 21:11:56.373288 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:11:56.372896    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8134-efc1e0" container="test-curl-8134-efc1e0"
Sep 10 21:11:56.393048 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.392977    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb" podStartSLOduration=1.992744721 podStartE2EDuration="2.392963024s" totalImagesPullingTime="400.218303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:56.391932481 +0000 UTC m=+803.394178829" watchObservedRunningTime="2026-09-10 21:11:56.392963024 +0000 UTC m=+803.395209396"
Sep 10 21:11:56.431769 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.431747    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:11:56.705201 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:56.705177    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8134-238617"]
Sep 10 21:11:56.706744 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:11:56.706714    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dd08870_d48e_4137_bb3d_b1bd3dc75308.slice/crio-b0fa90252752cbfbd1595d5f432dc2b52699aafcd23267869f1dcb71f25813e3 WatchSource:0}: Error finding container b0fa90252752cbfbd1595d5f432dc2b52699aafcd23267869f1dcb71f25813e3: Status 404 returned error can't find the container with id b0fa90252752cbfbd1595d5f432dc2b52699aafcd23267869f1dcb71f25813e3
Sep 10 21:11:57.380014 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:57.379973    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8134-238617" event={"ID":"2dd08870-d48e-4137-bb3d-b1bd3dc75308","Type":"ContainerStarted","Data":"b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67"}
Sep 10 21:11:57.380014 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:57.380022    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8134-238617" event={"ID":"2dd08870-d48e-4137-bb3d-b1bd3dc75308","Type":"ContainerStarted","Data":"b0fa90252752cbfbd1595d5f432dc2b52699aafcd23267869f1dcb71f25813e3"}
Sep 10 21:11:57.380548 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:57.380333    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:11:57.380548 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:57.380313    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8134-efc1e0" podUID="e719ed0d-97d8-49ac-b2da-c5200b349501" containerName="test-curl-8134-efc1e0" containerID="cri-o://f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42" gracePeriod=2
Sep 10 21:11:57.392916 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:57.392874    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8134-238617" podStartSLOduration=1.392858852 podStartE2EDuration="1.392858852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:11:57.392411059 +0000 UTC m=+804.394657406" watchObservedRunningTime="2026-09-10 21:11:57.392858852 +0000 UTC m=+804.395105202"
Sep 10 21:11:57.394024 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:11:57.393997    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8134-efc1e0\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="e719ed0d-97d8-49ac-b2da-c5200b349501" pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:11:57.975738 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:57.975704    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8134-238617"]
Sep 10 21:11:57.979861 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:57.979835    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8134-238617"]
Sep 10 21:11:58.386578 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:11:58.386546    2580 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8134-238617" container="test-curl-8134-238617"
Sep 10 21:11:58.854250 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:58.854176    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"]
Sep 10 21:11:58.854594 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:58.854572    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" containerName="test-curl-8134-238617"
Sep 10 21:11:58.854594 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:58.854594    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" containerName="test-curl-8134-238617"
Sep 10 21:11:58.854801 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:58.854680    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" containerName="test-curl-8134-238617"
Sep 10 21:11:58.857287 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:58.857267    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:58.859537 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:58.859520    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:11:58.859633 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:58.859520    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-5e2fa5-openshift-default-dockercfg-b42s4\""
Sep 10 21:11:58.868050 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:58.868028    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"]
Sep 10 21:11:59.014071 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014040    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-data\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.014071 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-podinfo\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.014343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014089    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-envoy\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.014343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014160    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-credential-socket\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.014343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014191    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-token\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.014343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014212    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzzw\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-kube-api-access-5bzzw\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.014343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014241    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc529b33-6703-43ba-8ea6-e58edd0c7689-istiod-ca-cert\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.014343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014300    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-socket\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.014343 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.014337    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-certs\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115357 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115260    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-data\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115357 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115313    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-podinfo\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115357 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115340    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-envoy\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115641 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115405    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-credential-socket\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115641 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115440    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-token\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115641 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115473    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzzw\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-kube-api-access-5bzzw\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115641 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115515    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc529b33-6703-43ba-8ea6-e58edd0c7689-istiod-ca-cert\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115641 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115549    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-socket\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115888 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115691    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-data\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115888 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115720    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-certs\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.115987 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.115899    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-socket\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.116166 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.116145    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-certs\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.116315 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.116244    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-credential-socket\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.116488 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.116461    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc529b33-6703-43ba-8ea6-e58edd0c7689-istiod-ca-cert\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.118046 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.118023    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-envoy\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.118208 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.118192    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-podinfo\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.123797 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.123776    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-token\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.124747 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.124724    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzzw\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-kube-api-access-5bzzw\") pod \"e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") " pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.168875 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.168843    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:11:59.391935 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.391878    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8134-238617" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" containerName="test-curl-8134-238617" containerID="cri-o://b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67" gracePeriod=2
Sep 10 21:11:59.393976 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:11:59.393949    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8134-238617\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:11:59.448363 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.448338    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"]
Sep 10 21:11:59.449963 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:11:59.449935    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc529b33_6703_43ba_8ea6_e58edd0c7689.slice/crio-90d20b6f56d791e867c87a823d989a62a14225187114c306d7e47fafe7ecd983 WatchSource:0}: Error finding container 90d20b6f56d791e867c87a823d989a62a14225187114c306d7e47fafe7ecd983: Status 404 returned error can't find the container with id 90d20b6f56d791e867c87a823d989a62a14225187114c306d7e47fafe7ecd983
Sep 10 21:11:59.564038 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.564015    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:11:59.722894 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.722815    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrv8f\" (UniqueName: \"kubernetes.io/projected/e719ed0d-97d8-49ac-b2da-c5200b349501-kube-api-access-xrv8f\") pod \"e719ed0d-97d8-49ac-b2da-c5200b349501\" (UID: \"e719ed0d-97d8-49ac-b2da-c5200b349501\") "
Sep 10 21:11:59.724972 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.724941    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e719ed0d-97d8-49ac-b2da-c5200b349501-kube-api-access-xrv8f" (OuterVolumeSpecName: "kube-api-access-xrv8f") pod "e719ed0d-97d8-49ac-b2da-c5200b349501" (UID: "e719ed0d-97d8-49ac-b2da-c5200b349501"). InnerVolumeSpecName "kube-api-access-xrv8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:11:59.824539 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:11:59.824499    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrv8f\" (UniqueName: \"kubernetes.io/projected/e719ed0d-97d8-49ac-b2da-c5200b349501-kube-api-access-xrv8f\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:12:00.403187 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:00.403075    2580 generic.go:356] "Generic (PLEG): container finished" podID="e719ed0d-97d8-49ac-b2da-c5200b349501" containerID="f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42" exitCode=137
Sep 10 21:12:00.403187 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:00.403176    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:12:00.403644 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:00.403225    2580 scope.go:117] "RemoveContainer" containerID="f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42"
Sep 10 21:12:00.405260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:00.405148    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" event={"ID":"fc529b33-6703-43ba-8ea6-e58edd0c7689","Type":"ContainerStarted","Data":"90d20b6f56d791e867c87a823d989a62a14225187114c306d7e47fafe7ecd983"}
Sep 10 21:12:00.413349 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:00.413324    2580 scope.go:117] "RemoveContainer" containerID="f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42"
Sep 10 21:12:00.413784 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:00.413636    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42\": container with ID starting with f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42 not found: ID does not exist" containerID="f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42"
Sep 10 21:12:00.413784 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:00.413672    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42"} err="failed to get container status \"f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42\": rpc error: code = NotFound desc = could not find container \"f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42\": container with ID starting with f94f4537778f960d6426435babf5de8578a83cafabb191a5b64c64ea0122fb42 not found: ID does not exist"
Sep 10 21:12:00.416515 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:00.416489    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8134-efc1e0\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="e719ed0d-97d8-49ac-b2da-c5200b349501" pod="opendatahub/test-curl-8134-efc1e0"
Sep 10 21:12:01.521454 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:01.521420    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e719ed0d-97d8-49ac-b2da-c5200b349501" path="/var/lib/kubelet/pods/e719ed0d-97d8-49ac-b2da-c5200b349501/volumes"
Sep 10 21:12:02.170982 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.170947    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:12:02.171065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.171044    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:12:02.171104 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.171076    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:12:02.267494 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.267471    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:12:02.420339 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.420303    2580 generic.go:356] "Generic (PLEG): container finished" podID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" containerID="b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67" exitCode=137
Sep 10 21:12:02.420514 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.420357    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:12:02.420514 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.420394    2580 scope.go:117] "RemoveContainer" containerID="b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67"
Sep 10 21:12:02.422382 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.422349    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" event={"ID":"fc529b33-6703-43ba-8ea6-e58edd0c7689","Type":"ContainerStarted","Data":"a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476"}
Sep 10 21:12:02.429807 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.429791    2580 scope.go:117] "RemoveContainer" containerID="b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67"
Sep 10 21:12:02.430090 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:02.430071    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67\": container with ID starting with b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67 not found: ID does not exist" containerID="b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67"
Sep 10 21:12:02.430179 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.430098    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67"} err="failed to get container status \"b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67\": rpc error: code = NotFound desc = could not find container \"b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67\": container with ID starting with b6521d0b7e5729a160e0fbfb066866b2dc9d2e68e36251623f86407aad2e6b67 not found: ID does not exist"
Sep 10 21:12:02.445563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.445523    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" podStartSLOduration=1.726986477 podStartE2EDuration="4.445510732s" totalImagesPullingTime="2.718524255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:02.44294459 +0000 UTC m=+809.445190937" watchObservedRunningTime="2026-09-10 21:12:02.445510732 +0000 UTC m=+809.447757078"
Sep 10 21:12:02.449082 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.449048    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncrw6\" (UniqueName: \"kubernetes.io/projected/2dd08870-d48e-4137-bb3d-b1bd3dc75308-kube-api-access-ncrw6\") pod \"2dd08870-d48e-4137-bb3d-b1bd3dc75308\" (UID: \"2dd08870-d48e-4137-bb3d-b1bd3dc75308\") "
Sep 10 21:12:02.451460 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.451433    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd08870-d48e-4137-bb3d-b1bd3dc75308-kube-api-access-ncrw6" (OuterVolumeSpecName: "kube-api-access-ncrw6") pod "2dd08870-d48e-4137-bb3d-b1bd3dc75308" (UID: "2dd08870-d48e-4137-bb3d-b1bd3dc75308"). InnerVolumeSpecName "kube-api-access-ncrw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:02.551026 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.550991    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncrw6\" (UniqueName: \"kubernetes.io/projected/2dd08870-d48e-4137-bb3d-b1bd3dc75308-kube-api-access-ncrw6\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:12:02.697636 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.697548    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"]
Sep 10 21:12:02.703056 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.703024    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.705437 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.705411    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8086f00e-serving-cert\""
Sep 10 21:12:02.715262 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.715235    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"]
Sep 10 21:12:02.733468 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:02.733435    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8134-238617\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:12:02.853780 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.853746    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5ddg\" (UniqueName: \"kubernetes.io/projected/6152018d-88fe-4ce2-a2ac-d169040f0735-kube-api-access-g5ddg\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.853780 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.853779    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-metrics-certs\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.854003 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.853801    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6152018d-88fe-4ce2-a2ac-d169040f0735-openshift-service-ca\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.854003 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.853863    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.954795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.954717    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5ddg\" (UniqueName: \"kubernetes.io/projected/6152018d-88fe-4ce2-a2ac-d169040f0735-kube-api-access-g5ddg\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.954795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.954748    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-metrics-certs\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.954795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.954770    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6152018d-88fe-4ce2-a2ac-d169040f0735-openshift-service-ca\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.954795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.954796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.955076 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:02.954921    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-serving-cert: secret "maas-api-e2e-ait-8086f00e-serving-cert" not found
Sep 10 21:12:02.955076 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:02.954974    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls podName:6152018d-88fe-4ce2-a2ac-d169040f0735 nodeName:}" failed. No retries permitted until 2026-09-10 21:12:03.454959226 +0000 UTC m=+810.457205551 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls") pod "maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh" (UID: "6152018d-88fe-4ce2-a2ac-d169040f0735") : secret "maas-api-e2e-ait-8086f00e-serving-cert" not found
Sep 10 21:12:02.955468 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.955448    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6152018d-88fe-4ce2-a2ac-d169040f0735-openshift-service-ca\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.957364 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.957342    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-metrics-certs\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:02.962711 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:02.962690    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5ddg\" (UniqueName: \"kubernetes.io/projected/6152018d-88fe-4ce2-a2ac-d169040f0735-kube-api-access-g5ddg\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:03.169361 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.169333    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:12:03.174474 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.174450    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:12:03.176198 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:03.176171    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8134-238617\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:12:03.392303 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.392273    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:12:03.410600 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:03.410567    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8134-238617\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:12:03.430313 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.430286    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:12:03.431345 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.431322    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:12:03.455995 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:03.455954    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8134-238617\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:12:03.459622 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.459450    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:03.462597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.462570    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls\") pod \"maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:03.517662 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:03.517615    2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8134-238617\" is forbidden: User \"system:node:ip-10-0-128-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-236.ec2.internal' and this object" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" pod="opendatahub/test-curl-8134-238617"
Sep 10 21:12:03.520209 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.520185    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd08870-d48e-4137-bb3d-b1bd3dc75308" path="/var/lib/kubelet/pods/2dd08870-d48e-4137-bb3d-b1bd3dc75308/volumes"
Sep 10 21:12:03.616786 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.616755    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:03.894875 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:03.894849    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"]
Sep 10 21:12:03.897941 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:12:03.897905    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6152018d_88fe_4ce2_a2ac_d169040f0735.slice/crio-cbc5075829f8a23c8f04cdb8ce5fbef95cf0e348c31a1829129f063f5ef91ba5 WatchSource:0}: Error finding container cbc5075829f8a23c8f04cdb8ce5fbef95cf0e348c31a1829129f063f5ef91ba5: Status 404 returned error can't find the container with id cbc5075829f8a23c8f04cdb8ce5fbef95cf0e348c31a1829129f063f5ef91ba5
Sep 10 21:12:04.438170 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:04.438103    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh" event={"ID":"6152018d-88fe-4ce2-a2ac-d169040f0735","Type":"ContainerStarted","Data":"3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb"}
Sep 10 21:12:04.438394 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:04.438203    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh" event={"ID":"6152018d-88fe-4ce2-a2ac-d169040f0735","Type":"ContainerStarted","Data":"cbc5075829f8a23c8f04cdb8ce5fbef95cf0e348c31a1829129f063f5ef91ba5"}
Sep 10 21:12:04.438557 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:04.438533    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:04.458281 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:04.458237    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh" podStartSLOduration=2.042470184 podStartE2EDuration="2.458222763s" totalImagesPullingTime="415.752579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:04.456507867 +0000 UTC m=+811.458754216" watchObservedRunningTime="2026-09-10 21:12:04.458222763 +0000 UTC m=+811.460469111"
Sep 10 21:12:05.506240 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.506209    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"]
Sep 10 21:12:05.511091 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.511070    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.513603 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.513215    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-5e2fa5-serving-cert\""
Sep 10 21:12:05.522590 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.522562    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"]
Sep 10 21:12:05.678597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.678563    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-maas-api-tls\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.678796 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.678623    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-metrics-certs\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.678796 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.678738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4925272e-744b-4ff9-a0ee-3571615d3b69-openshift-service-ca\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.678796 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.678772    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z5bh\" (UniqueName: \"kubernetes.io/projected/4925272e-744b-4ff9-a0ee-3571615d3b69-kube-api-access-8z5bh\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.779459 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.779373    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-maas-api-tls\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.779459 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.779431    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-metrics-certs\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.779678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.779505    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4925272e-744b-4ff9-a0ee-3571615d3b69-openshift-service-ca\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.779678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.779537    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z5bh\" (UniqueName: \"kubernetes.io/projected/4925272e-744b-4ff9-a0ee-3571615d3b69-kube-api-access-8z5bh\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.780444 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.780412    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4925272e-744b-4ff9-a0ee-3571615d3b69-openshift-service-ca\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.782015 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.781988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-maas-api-tls\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.782158 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.782096    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-metrics-certs\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.787580 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.787560    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z5bh\" (UniqueName: \"kubernetes.io/projected/4925272e-744b-4ff9-a0ee-3571615d3b69-kube-api-access-8z5bh\") pod \"maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:05.827704 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:05.827665    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:06.119591 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.119558    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"]
Sep 10 21:12:06.121372 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:12:06.121330    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4925272e_744b_4ff9_a0ee_3571615d3b69.slice/crio-0628152af193f5d91bf90ea65ee6aa6e2dc7a497e7fda5fb913a3ed5630413d3 WatchSource:0}: Error finding container 0628152af193f5d91bf90ea65ee6aa6e2dc7a497e7fda5fb913a3ed5630413d3: Status 404 returned error can't find the container with id 0628152af193f5d91bf90ea65ee6aa6e2dc7a497e7fda5fb913a3ed5630413d3
Sep 10 21:12:06.285251 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.285218    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"]
Sep 10 21:12:06.290314 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.290296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:06.292368 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.292342    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5e2fa5\""
Sep 10 21:12:06.292468 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.292382    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-5e2fa5-dockercfg-dv6jg\""
Sep 10 21:12:06.297029 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.297007    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"]
Sep 10 21:12:06.388902 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.388869    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5z8p\" (UniqueName: \"kubernetes.io/projected/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-kube-api-access-r5z8p\") pod \"payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z\" (UID: \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:06.388902 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.388903    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z\" (UID: \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:06.451733 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.451703    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v" event={"ID":"4925272e-744b-4ff9-a0ee-3571615d3b69","Type":"ContainerStarted","Data":"0628152af193f5d91bf90ea65ee6aa6e2dc7a497e7fda5fb913a3ed5630413d3"}
Sep 10 21:12:06.489637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.489608    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5z8p\" (UniqueName: \"kubernetes.io/projected/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-kube-api-access-r5z8p\") pod \"payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z\" (UID: \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:06.489736 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.489657    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z\" (UID: \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:06.490507 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.490484    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z\" (UID: \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:06.497235 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.497214    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5z8p\" (UniqueName: \"kubernetes.io/projected/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-kube-api-access-r5z8p\") pod \"payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z\" (UID: \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:06.603171 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.603133    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:06.880294 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:06.880264    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"]
Sep 10 21:12:06.881510 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:12:06.881481    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff909dd7_d6a4_4687_8a8b_2d7783dd1c5c.slice/crio-24f739ee96f26c9050ad5dc8ec7ebe26b6d5252c82aa0525d4b747e84f232503 WatchSource:0}: Error finding container 24f739ee96f26c9050ad5dc8ec7ebe26b6d5252c82aa0525d4b747e84f232503: Status 404 returned error can't find the container with id 24f739ee96f26c9050ad5dc8ec7ebe26b6d5252c82aa0525d4b747e84f232503
Sep 10 21:12:07.459492 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:07.459454    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v" event={"ID":"4925272e-744b-4ff9-a0ee-3571615d3b69","Type":"ContainerStarted","Data":"73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456"}
Sep 10 21:12:07.459693 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:07.459635    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:07.460935 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:07.460915    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z" event={"ID":"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c","Type":"ContainerStarted","Data":"97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af"}
Sep 10 21:12:07.461032 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:07.460950    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z" event={"ID":"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c","Type":"ContainerStarted","Data":"24f739ee96f26c9050ad5dc8ec7ebe26b6d5252c82aa0525d4b747e84f232503"}
Sep 10 21:12:07.461032 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:07.461005    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:07.497775 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:07.497731    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z" podStartSLOduration=1.4977162800000001 podStartE2EDuration="1.49771628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:07.496852303 +0000 UTC m=+814.499098653" watchObservedRunningTime="2026-09-10 21:12:07.49771628 +0000 UTC m=+814.499962627"
Sep 10 21:12:07.499383 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:07.499352    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v" podStartSLOduration=2.153898168 podStartE2EDuration="2.499344251s" totalImagesPullingTime="345.446083ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:07.478145101 +0000 UTC m=+814.480391443" watchObservedRunningTime="2026-09-10 21:12:07.499344251 +0000 UTC m=+814.501590597"
Sep 10 21:12:10.449494 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:10.449462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:13.471857 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:13.471830    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:12:18.467961 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:18.467929    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:12:23.435260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.435222    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"]
Sep 10 21:12:23.447509 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.447484    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.449495 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.449468    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"]
Sep 10 21:12:23.449796 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.449780    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-414d5f-serving-cert\""
Sep 10 21:12:23.552315 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.552279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-metrics-certs\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.552486 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.552348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa77cb97-c9cc-455f-913a-437ed371db23-openshift-service-ca\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.552486 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.552401    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xmhf\" (UniqueName: \"kubernetes.io/projected/aa77cb97-c9cc-455f-913a-437ed371db23-kube-api-access-5xmhf\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.552486 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.552438    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.660881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.653536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-metrics-certs\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.660881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.653609    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa77cb97-c9cc-455f-913a-437ed371db23-openshift-service-ca\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.660881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.653674    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xmhf\" (UniqueName: \"kubernetes.io/projected/aa77cb97-c9cc-455f-913a-437ed371db23-kube-api-access-5xmhf\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.660881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.653718    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.660881 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:23.653902    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-serving-cert: secret "maas-api-e2e-inf-b-414d5f-serving-cert" not found
Sep 10 21:12:23.660881 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:23.654016    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls podName:aa77cb97-c9cc-455f-913a-437ed371db23 nodeName:}" failed. No retries permitted until 2026-09-10 21:12:24.153994261 +0000 UTC m=+831.156240603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls") pod "maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg" (UID: "aa77cb97-c9cc-455f-913a-437ed371db23") : secret "maas-api-e2e-inf-b-414d5f-serving-cert" not found
Sep 10 21:12:23.660881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.654380    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa77cb97-c9cc-455f-913a-437ed371db23-openshift-service-ca\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.660881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.656025    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-metrics-certs\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:23.670275 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:23.670256    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xmhf\" (UniqueName: \"kubernetes.io/projected/aa77cb97-c9cc-455f-913a-437ed371db23-kube-api-access-5xmhf\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:24.159585 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.159548    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:24.162089 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.162056    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls\") pod \"maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:24.362637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.362602    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:24.562847 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.562809    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerID="a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476" exitCode=0
Sep 10 21:12:24.563215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.562871    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" event={"ID":"fc529b33-6703-43ba-8ea6-e58edd0c7689","Type":"ContainerDied","Data":"a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476"}
Sep 10 21:12:24.563260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.563234    2580 scope.go:117] "RemoveContainer" containerID="a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476"
Sep 10 21:12:24.564313 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.564287    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:12:24.564406 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.564346    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:12:24.564406 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.564379    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:12:24.636418 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:24.636396    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"]
Sep 10 21:12:24.637691 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:12:24.637665    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa77cb97_c9cc_455f_913a_437ed371db23.slice/crio-714bc9aa78a88c731bac2e563d1b300dda49623b7d1c317c5391c6b0dff57aca WatchSource:0}: Error finding container 714bc9aa78a88c731bac2e563d1b300dda49623b7d1c317c5391c6b0dff57aca: Status 404 returned error can't find the container with id 714bc9aa78a88c731bac2e563d1b300dda49623b7d1c317c5391c6b0dff57aca
Sep 10 21:12:25.570101 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:25.570062    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" event={"ID":"fc529b33-6703-43ba-8ea6-e58edd0c7689","Type":"ContainerStarted","Data":"5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af"}
Sep 10 21:12:25.571874 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:25.571846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg" event={"ID":"aa77cb97-c9cc-455f-913a-437ed371db23","Type":"ContainerStarted","Data":"104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d"}
Sep 10 21:12:25.571986 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:25.571896    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg" event={"ID":"aa77cb97-c9cc-455f-913a-437ed371db23","Type":"ContainerStarted","Data":"714bc9aa78a88c731bac2e563d1b300dda49623b7d1c317c5391c6b0dff57aca"}
Sep 10 21:12:25.571986 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:25.571961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:25.612022 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:25.611965    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg" podStartSLOduration=2.348359339 podStartE2EDuration="2.611949705s" totalImagesPullingTime="263.590366ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:25.609216117 +0000 UTC m=+832.611462461" watchObservedRunningTime="2026-09-10 21:12:25.611949705 +0000 UTC m=+832.614196052"
Sep 10 21:12:26.169246 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:26.169205    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:12:26.169459 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:26.169438    2580 patch_prober.go:28] interesting pod/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:12:26.169512 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:26.169493    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:12:27.169498 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:27.169454    2580 patch_prober.go:28] interesting pod/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:12:27.169890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:27.169535    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:12:28.170048 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:28.170013    2580 patch_prober.go:28] interesting pod/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:12:28.170467 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:28.170083    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:12:29.169803 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:29.169764    2580 patch_prober.go:28] interesting pod/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:12:29.169993 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:29.169823    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:12:30.169236 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:30.169199    2580 patch_prober.go:28] interesting pod/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 10 21:12:30.169634 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:30.169262    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 10 21:12:31.173700 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:31.173670    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:12:31.174069 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:31.173941    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:12:31.174633 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:31.174616    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:12:31.585076 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:31.585001    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:12:43.702462 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:43.702424    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"]
Sep 10 21:12:43.702986 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:43.702784    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh" podUID="6152018d-88fe-4ce2-a2ac-d169040f0735" containerName="maas-api" containerID="cri-o://3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb" gracePeriod=30
Sep 10 21:12:43.951753 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:43.951730    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:44.053440 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.053360    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-metrics-certs\") pod \"6152018d-88fe-4ce2-a2ac-d169040f0735\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") "
Sep 10 21:12:44.053440 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.053412    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5ddg\" (UniqueName: \"kubernetes.io/projected/6152018d-88fe-4ce2-a2ac-d169040f0735-kube-api-access-g5ddg\") pod \"6152018d-88fe-4ce2-a2ac-d169040f0735\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") "
Sep 10 21:12:44.053616 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.053473    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls\") pod \"6152018d-88fe-4ce2-a2ac-d169040f0735\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") "
Sep 10 21:12:44.053616 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.053515    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6152018d-88fe-4ce2-a2ac-d169040f0735-openshift-service-ca\") pod \"6152018d-88fe-4ce2-a2ac-d169040f0735\" (UID: \"6152018d-88fe-4ce2-a2ac-d169040f0735\") "
Sep 10 21:12:44.053989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.053953    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6152018d-88fe-4ce2-a2ac-d169040f0735-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6152018d-88fe-4ce2-a2ac-d169040f0735" (UID: "6152018d-88fe-4ce2-a2ac-d169040f0735"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:12:44.056524 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.056499    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6152018d-88fe-4ce2-a2ac-d169040f0735" (UID: "6152018d-88fe-4ce2-a2ac-d169040f0735"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:12:44.057755 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.057723    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6152018d-88fe-4ce2-a2ac-d169040f0735" (UID: "6152018d-88fe-4ce2-a2ac-d169040f0735"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:12:44.057755 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.057729    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6152018d-88fe-4ce2-a2ac-d169040f0735-kube-api-access-g5ddg" (OuterVolumeSpecName: "kube-api-access-g5ddg") pod "6152018d-88fe-4ce2-a2ac-d169040f0735" (UID: "6152018d-88fe-4ce2-a2ac-d169040f0735"). InnerVolumeSpecName "kube-api-access-g5ddg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:12:44.155254 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.155227    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:12:44.155254 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.155253    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6152018d-88fe-4ce2-a2ac-d169040f0735-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:12:44.155446 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.155263    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6152018d-88fe-4ce2-a2ac-d169040f0735-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:12:44.155446 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.155271    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5ddg\" (UniqueName: \"kubernetes.io/projected/6152018d-88fe-4ce2-a2ac-d169040f0735-kube-api-access-g5ddg\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:12:44.718001 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.717960    2580 generic.go:356] "Generic (PLEG): container finished" podID="6152018d-88fe-4ce2-a2ac-d169040f0735" containerID="3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb" exitCode=0
Sep 10 21:12:44.718425 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.718009    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"
Sep 10 21:12:44.718425 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.718040    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh" event={"ID":"6152018d-88fe-4ce2-a2ac-d169040f0735","Type":"ContainerDied","Data":"3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb"}
Sep 10 21:12:44.718425 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.718093    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh" event={"ID":"6152018d-88fe-4ce2-a2ac-d169040f0735","Type":"ContainerDied","Data":"cbc5075829f8a23c8f04cdb8ce5fbef95cf0e348c31a1829129f063f5ef91ba5"}
Sep 10 21:12:44.718425 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.718140    2580 scope.go:117] "RemoveContainer" containerID="3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb"
Sep 10 21:12:44.728318 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.728301    2580 scope.go:117] "RemoveContainer" containerID="3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb"
Sep 10 21:12:44.728625 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:44.728606    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb\": container with ID starting with 3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb not found: ID does not exist" containerID="3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb"
Sep 10 21:12:44.728677 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.728635    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb"} err="failed to get container status \"3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb\": rpc error: code = NotFound desc = could not find container \"3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb\": container with ID starting with 3afa5cc7e45fbd0a0a5264862630b13fbc93d394878321c2863f265675c685bb not found: ID does not exist"
Sep 10 21:12:44.739067 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.739041    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"]
Sep 10 21:12:44.742538 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:44.742520    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8086f00e-749b98f8fc-9kfjh"]
Sep 10 21:12:45.519064 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:45.519033    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6152018d-88fe-4ce2-a2ac-d169040f0735" path="/var/lib/kubelet/pods/6152018d-88fe-4ce2-a2ac-d169040f0735/volumes"
Sep 10 21:12:48.243556 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.243523    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"]
Sep 10 21:12:48.243976 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.243963    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6152018d-88fe-4ce2-a2ac-d169040f0735" containerName="maas-api"
Sep 10 21:12:48.244020 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.243978    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6152018d-88fe-4ce2-a2ac-d169040f0735" containerName="maas-api"
Sep 10 21:12:48.244072 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.244063    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6152018d-88fe-4ce2-a2ac-d169040f0735" containerName="maas-api"
Sep 10 21:12:48.249116 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.249093    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.251087 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.251068    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-065d3bbf-serving-cert\""
Sep 10 21:12:48.259866 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.259845    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"]
Sep 10 21:12:48.394956 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.394924    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.395195 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.394982    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5q4d\" (UniqueName: \"kubernetes.io/projected/863a40e6-d41c-47bb-90d0-9db83cf28e59-kube-api-access-l5q4d\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.395195 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.395049    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863a40e6-d41c-47bb-90d0-9db83cf28e59-openshift-service-ca\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.395195 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.395152    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-metrics-certs\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.496139 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.496038    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.496283 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.496111    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5q4d\" (UniqueName: \"kubernetes.io/projected/863a40e6-d41c-47bb-90d0-9db83cf28e59-kube-api-access-l5q4d\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.496283 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.496173    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863a40e6-d41c-47bb-90d0-9db83cf28e59-openshift-service-ca\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.496283 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.496223    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-metrics-certs\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.496388 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:48.496225    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-serving-cert: secret "maas-api-e2e-migrate-065d3bbf-serving-cert" not found
Sep 10 21:12:48.496388 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:12:48.496366    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls podName:863a40e6-d41c-47bb-90d0-9db83cf28e59 nodeName:}" failed. No retries permitted until 2026-09-10 21:12:48.99634333 +0000 UTC m=+855.998589662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls") pod "maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx" (UID: "863a40e6-d41c-47bb-90d0-9db83cf28e59") : secret "maas-api-e2e-migrate-065d3bbf-serving-cert" not found
Sep 10 21:12:48.496908 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.496844    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863a40e6-d41c-47bb-90d0-9db83cf28e59-openshift-service-ca\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.498707 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.498685    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-metrics-certs\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.504580 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.504555    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5q4d\" (UniqueName: \"kubernetes.io/projected/863a40e6-d41c-47bb-90d0-9db83cf28e59-kube-api-access-l5q4d\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:48.848447 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.848372    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"]
Sep 10 21:12:48.852028 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.852012    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:48.854034 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.854003    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-065d3bbf\""
Sep 10 21:12:48.854204 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.854087    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-065d3bbf-dockercfg-4wx7n\""
Sep 10 21:12:48.862058 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:48.862031    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"]
Sep 10 21:12:49.000814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.000778    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:49.000964 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.000843    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebc612-0cc8-4d83-8140-82aa622524d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx\" (UID: \"a1ebc612-0cc8-4d83-8140-82aa622524d1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:49.000964 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.000883    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp9bz\" (UniqueName: \"kubernetes.io/projected/a1ebc612-0cc8-4d83-8140-82aa622524d1-kube-api-access-gp9bz\") pod \"payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx\" (UID: \"a1ebc612-0cc8-4d83-8140-82aa622524d1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:49.003684 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.003649    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls\") pod \"maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:49.101710 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.101618    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebc612-0cc8-4d83-8140-82aa622524d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx\" (UID: \"a1ebc612-0cc8-4d83-8140-82aa622524d1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:49.101710 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.101659    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp9bz\" (UniqueName: \"kubernetes.io/projected/a1ebc612-0cc8-4d83-8140-82aa622524d1-kube-api-access-gp9bz\") pod \"payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx\" (UID: \"a1ebc612-0cc8-4d83-8140-82aa622524d1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:49.102423 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.102393    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebc612-0cc8-4d83-8140-82aa622524d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx\" (UID: \"a1ebc612-0cc8-4d83-8140-82aa622524d1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:49.111715 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.111693    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp9bz\" (UniqueName: \"kubernetes.io/projected/a1ebc612-0cc8-4d83-8140-82aa622524d1-kube-api-access-gp9bz\") pod \"payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx\" (UID: \"a1ebc612-0cc8-4d83-8140-82aa622524d1\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:49.161742 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.161710    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:49.162732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.162712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:49.442041 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.442016    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"]
Sep 10 21:12:49.443355 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:12:49.443332    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod863a40e6_d41c_47bb_90d0_9db83cf28e59.slice/crio-df61d4144f835a3e98b42239f75649fcfacf983c1e480e46b303a61a37267f1b WatchSource:0}: Error finding container df61d4144f835a3e98b42239f75649fcfacf983c1e480e46b303a61a37267f1b: Status 404 returned error can't find the container with id df61d4144f835a3e98b42239f75649fcfacf983c1e480e46b303a61a37267f1b
Sep 10 21:12:49.451280 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.451256    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"]
Sep 10 21:12:49.452191 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:12:49.452164    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ebc612_0cc8_4d83_8140_82aa622524d1.slice/crio-5d89e9b4c3377000dcab3ea7d61d775c1c7672f1e2aedbe2a8119faf5bfc5fe7 WatchSource:0}: Error finding container 5d89e9b4c3377000dcab3ea7d61d775c1c7672f1e2aedbe2a8119faf5bfc5fe7: Status 404 returned error can't find the container with id 5d89e9b4c3377000dcab3ea7d61d775c1c7672f1e2aedbe2a8119faf5bfc5fe7
Sep 10 21:12:49.749387 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.749289    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx" event={"ID":"a1ebc612-0cc8-4d83-8140-82aa622524d1","Type":"ContainerStarted","Data":"58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced"}
Sep 10 21:12:49.749387 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.749358    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:12:49.749387 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.749383    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx" event={"ID":"a1ebc612-0cc8-4d83-8140-82aa622524d1","Type":"ContainerStarted","Data":"5d89e9b4c3377000dcab3ea7d61d775c1c7672f1e2aedbe2a8119faf5bfc5fe7"}
Sep 10 21:12:49.750624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.750601    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx" event={"ID":"863a40e6-d41c-47bb-90d0-9db83cf28e59","Type":"ContainerStarted","Data":"df61d4144f835a3e98b42239f75649fcfacf983c1e480e46b303a61a37267f1b"}
Sep 10 21:12:49.769085 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:49.769039    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx" podStartSLOduration=1.769022973 podStartE2EDuration="1.769022973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:49.766216622 +0000 UTC m=+856.768462973" watchObservedRunningTime="2026-09-10 21:12:49.769022973 +0000 UTC m=+856.771269370"
Sep 10 21:12:50.757939 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:50.757907    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx" event={"ID":"863a40e6-d41c-47bb-90d0-9db83cf28e59","Type":"ContainerStarted","Data":"c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0"}
Sep 10 21:12:50.758375 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:50.757975    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:56.768837 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:56.768806    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:12:56.788507 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:12:56.788456    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx" podStartSLOduration=8.331728213 podStartE2EDuration="8.788440459s" totalImagesPullingTime="456.712246ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:12:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:12:50.779676957 +0000 UTC m=+857.781923303" watchObservedRunningTime="2026-09-10 21:12:56.788440459 +0000 UTC m=+863.790686806"
Sep 10 21:13:00.759901 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:00.759866    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:13:19.555552 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.555518    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"]
Sep 10 21:13:19.555991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.555758    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx" podUID="863a40e6-d41c-47bb-90d0-9db83cf28e59" containerName="maas-api" containerID="cri-o://c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0" gracePeriod=30
Sep 10 21:13:19.618390 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.618356    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"]
Sep 10 21:13:19.618704 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.618672    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx" podUID="a1ebc612-0cc8-4d83-8140-82aa622524d1" containerName="payload-pre-processing" containerID="cri-o://58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced" gracePeriod=30
Sep 10 21:13:19.881561 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.881293    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:13:19.911825 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.911792    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:13:19.932829 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.932789    2580 generic.go:356] "Generic (PLEG): container finished" podID="a1ebc612-0cc8-4d83-8140-82aa622524d1" containerID="58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced" exitCode=0
Sep 10 21:13:19.932991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.932844    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"
Sep 10 21:13:19.932991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.932848    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx" event={"ID":"a1ebc612-0cc8-4d83-8140-82aa622524d1","Type":"ContainerDied","Data":"58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced"}
Sep 10 21:13:19.932991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.932923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx" event={"ID":"a1ebc612-0cc8-4d83-8140-82aa622524d1","Type":"ContainerDied","Data":"5d89e9b4c3377000dcab3ea7d61d775c1c7672f1e2aedbe2a8119faf5bfc5fe7"}
Sep 10 21:13:19.932991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.932954    2580 scope.go:117] "RemoveContainer" containerID="58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced"
Sep 10 21:13:19.934907 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.934881    2580 generic.go:356] "Generic (PLEG): container finished" podID="863a40e6-d41c-47bb-90d0-9db83cf28e59" containerID="c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0" exitCode=0
Sep 10 21:13:19.935011 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.934935    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx" event={"ID":"863a40e6-d41c-47bb-90d0-9db83cf28e59","Type":"ContainerDied","Data":"c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0"}
Sep 10 21:13:19.935011 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.934948    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"
Sep 10 21:13:19.935011 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.934977    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx" event={"ID":"863a40e6-d41c-47bb-90d0-9db83cf28e59","Type":"ContainerDied","Data":"df61d4144f835a3e98b42239f75649fcfacf983c1e480e46b303a61a37267f1b"}
Sep 10 21:13:19.943236 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.943212    2580 scope.go:117] "RemoveContainer" containerID="58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced"
Sep 10 21:13:19.946747 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:19.946714    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced\": container with ID starting with 58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced not found: ID does not exist" containerID="58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced"
Sep 10 21:13:19.946842 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.946757    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced"} err="failed to get container status \"58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced\": rpc error: code = NotFound desc = could not find container \"58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced\": container with ID starting with 58b97897d3374750dd2a0a04192cfe3e8b54ac05ad132b58792762ef82090ced not found: ID does not exist"
Sep 10 21:13:19.946842 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.946778    2580 scope.go:117] "RemoveContainer" containerID="c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0"
Sep 10 21:13:19.956272 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.956255    2580 scope.go:117] "RemoveContainer" containerID="c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0"
Sep 10 21:13:19.956532 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:19.956513    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0\": container with ID starting with c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0 not found: ID does not exist" containerID="c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0"
Sep 10 21:13:19.956580 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.956537    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0"} err="failed to get container status \"c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0\": rpc error: code = NotFound desc = could not find container \"c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0\": container with ID starting with c31c4e3f5d8760ae8cc88a9ac50fff03965b66a34afacc11533d11a1f4fc70b0 not found: ID does not exist"
Sep 10 21:13:19.972027 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.972003    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-metrics-certs\") pod \"863a40e6-d41c-47bb-90d0-9db83cf28e59\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") "
Sep 10 21:13:19.972141 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.972059    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5q4d\" (UniqueName: \"kubernetes.io/projected/863a40e6-d41c-47bb-90d0-9db83cf28e59-kube-api-access-l5q4d\") pod \"863a40e6-d41c-47bb-90d0-9db83cf28e59\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") "
Sep 10 21:13:19.972141 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.972094    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls\") pod \"863a40e6-d41c-47bb-90d0-9db83cf28e59\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") "
Sep 10 21:13:19.972141 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.972136    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp9bz\" (UniqueName: \"kubernetes.io/projected/a1ebc612-0cc8-4d83-8140-82aa622524d1-kube-api-access-gp9bz\") pod \"a1ebc612-0cc8-4d83-8140-82aa622524d1\" (UID: \"a1ebc612-0cc8-4d83-8140-82aa622524d1\") "
Sep 10 21:13:19.972314 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.972167    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863a40e6-d41c-47bb-90d0-9db83cf28e59-openshift-service-ca\") pod \"863a40e6-d41c-47bb-90d0-9db83cf28e59\" (UID: \"863a40e6-d41c-47bb-90d0-9db83cf28e59\") "
Sep 10 21:13:19.972314 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.972252    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebc612-0cc8-4d83-8140-82aa622524d1-plugins-config-volume\") pod \"a1ebc612-0cc8-4d83-8140-82aa622524d1\" (UID: \"a1ebc612-0cc8-4d83-8140-82aa622524d1\") "
Sep 10 21:13:19.972596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.972569    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863a40e6-d41c-47bb-90d0-9db83cf28e59-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "863a40e6-d41c-47bb-90d0-9db83cf28e59" (UID: "863a40e6-d41c-47bb-90d0-9db83cf28e59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:19.972861 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.972661    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ebc612-0cc8-4d83-8140-82aa622524d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a1ebc612-0cc8-4d83-8140-82aa622524d1" (UID: "a1ebc612-0cc8-4d83-8140-82aa622524d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:19.974289 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.974268    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ebc612-0cc8-4d83-8140-82aa622524d1-kube-api-access-gp9bz" (OuterVolumeSpecName: "kube-api-access-gp9bz") pod "a1ebc612-0cc8-4d83-8140-82aa622524d1" (UID: "a1ebc612-0cc8-4d83-8140-82aa622524d1"). InnerVolumeSpecName "kube-api-access-gp9bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:19.974289 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.974280    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "863a40e6-d41c-47bb-90d0-9db83cf28e59" (UID: "863a40e6-d41c-47bb-90d0-9db83cf28e59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:19.974431 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.974320    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863a40e6-d41c-47bb-90d0-9db83cf28e59-kube-api-access-l5q4d" (OuterVolumeSpecName: "kube-api-access-l5q4d") pod "863a40e6-d41c-47bb-90d0-9db83cf28e59" (UID: "863a40e6-d41c-47bb-90d0-9db83cf28e59"). InnerVolumeSpecName "kube-api-access-l5q4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:19.974431 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:19.974332    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "863a40e6-d41c-47bb-90d0-9db83cf28e59" (UID: "863a40e6-d41c-47bb-90d0-9db83cf28e59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:20.073259 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.073231    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:20.073259 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.073256    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5q4d\" (UniqueName: \"kubernetes.io/projected/863a40e6-d41c-47bb-90d0-9db83cf28e59-kube-api-access-l5q4d\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:20.073446 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.073266    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/863a40e6-d41c-47bb-90d0-9db83cf28e59-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:20.073446 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.073276    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp9bz\" (UniqueName: \"kubernetes.io/projected/a1ebc612-0cc8-4d83-8140-82aa622524d1-kube-api-access-gp9bz\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:20.073446 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.073284    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/863a40e6-d41c-47bb-90d0-9db83cf28e59-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:20.073446 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.073293    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ebc612-0cc8-4d83-8140-82aa622524d1-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:20.255165 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.255138    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"]
Sep 10 21:13:20.259533 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.259509    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-065d3bbf-846897c846-v6qxx"]
Sep 10 21:13:20.269157 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.269133    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"]
Sep 10 21:13:20.272919 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.272898    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-065d3bbf-94c47b59f-6zcvx"]
Sep 10 21:13:20.679132 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.679073    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"]
Sep 10 21:13:20.679653 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.679633    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="863a40e6-d41c-47bb-90d0-9db83cf28e59" containerName="maas-api"
Sep 10 21:13:20.679743 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.679655    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="863a40e6-d41c-47bb-90d0-9db83cf28e59" containerName="maas-api"
Sep 10 21:13:20.679743 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.679682    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1ebc612-0cc8-4d83-8140-82aa622524d1" containerName="payload-pre-processing"
Sep 10 21:13:20.679743 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.679691    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1ebc612-0cc8-4d83-8140-82aa622524d1" containerName="payload-pre-processing"
Sep 10 21:13:20.679898 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.679803    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="863a40e6-d41c-47bb-90d0-9db83cf28e59" containerName="maas-api"
Sep 10 21:13:20.679898 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.679819    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1ebc612-0cc8-4d83-8140-82aa622524d1" containerName="payload-pre-processing"
Sep 10 21:13:20.684733 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.684711    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.687413 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.687354    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-169213ba-openshift-default-dockercfg-p8x5k\""
Sep 10 21:13:20.696401 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.696378    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"]
Sep 10 21:13:20.780482 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780440    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-socket\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.780640 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780497    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-token\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.780640 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-podinfo\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.780640 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780592    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-credential-socket\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.780640 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780624    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-envoy\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.780869 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780658    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-certs\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.780869 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780684    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpwb\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-kube-api-access-xfpwb\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.780869 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780726    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istiod-ca-cert\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.780869 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.780783    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-data\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882014 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.881979    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-envoy\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882022    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-certs\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882050    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpwb\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-kube-api-access-xfpwb\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882090    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istiod-ca-cert\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882135    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-data\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882451 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882219    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-socket\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882451 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882250    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-token\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882451 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882280    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-podinfo\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882451 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882333    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-credential-socket\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882666 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882473    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-certs\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882721 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882677    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-credential-socket\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.882910 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882875    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-socket\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.883043 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882915    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-data\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.883043 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.882953    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istiod-ca-cert\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.884856 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.884830    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-podinfo\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.884984 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.884965    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-envoy\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.890602 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.890575    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpwb\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-kube-api-access-xfpwb\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.890690 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.890641    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-token\") pod \"e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") " pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:20.996626 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:20.996598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:21.275466 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.275436    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"]
Sep 10 21:13:21.276091 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:13:21.276066    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34aaaf9a_378f_4ce0_9936_d53fd9e4b099.slice/crio-61066c2940f30f56c6bf88288dd711dfaa0ab4b1e3ecb0ee9b53a865fc15f0dc WatchSource:0}: Error finding container 61066c2940f30f56c6bf88288dd711dfaa0ab4b1e3ecb0ee9b53a865fc15f0dc: Status 404 returned error can't find the container with id 61066c2940f30f56c6bf88288dd711dfaa0ab4b1e3ecb0ee9b53a865fc15f0dc
Sep 10 21:13:21.278403 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.278368    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:21.278486 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.278458    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:21.278532 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.278499    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:21.522731 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.522636    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="863a40e6-d41c-47bb-90d0-9db83cf28e59" path="/var/lib/kubelet/pods/863a40e6-d41c-47bb-90d0-9db83cf28e59/volumes"
Sep 10 21:13:21.523212 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.523191    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1ebc612-0cc8-4d83-8140-82aa622524d1" path="/var/lib/kubelet/pods/a1ebc612-0cc8-4d83-8140-82aa622524d1/volumes"
Sep 10 21:13:21.949221 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.949176    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" event={"ID":"34aaaf9a-378f-4ce0-9936-d53fd9e4b099","Type":"ContainerStarted","Data":"79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f"}
Sep 10 21:13:21.949890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.949248    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" event={"ID":"34aaaf9a-378f-4ce0-9936-d53fd9e4b099","Type":"ContainerStarted","Data":"61066c2940f30f56c6bf88288dd711dfaa0ab4b1e3ecb0ee9b53a865fc15f0dc"}
Sep 10 21:13:21.972728 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.972676    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" podStartSLOduration=1.9726619589999999 podStartE2EDuration="1.972661959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:21.970073565 +0000 UTC m=+888.972319913" watchObservedRunningTime="2026-09-10 21:13:21.972661959 +0000 UTC m=+888.974908307"
Sep 10 21:13:21.997215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:21.997184    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:23.002563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:23.002535    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:23.968102 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:23.968070    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:23.969085 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:23.969061    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:24.441072 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.441036    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"]
Sep 10 21:13:24.446600 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.446577    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.448643 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.448625    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-169213ba-serving-cert\""
Sep 10 21:13:24.457326 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.457206    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"]
Sep 10 21:13:24.515510 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.515429    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.515510 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.515473    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vpv6\" (UniqueName: \"kubernetes.io/projected/0e985862-f6c6-4a53-b85a-a395b3177e04-kube-api-access-4vpv6\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.515510 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.515498    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e985862-f6c6-4a53-b85a-a395b3177e04-openshift-service-ca\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.515777 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.515647    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-metrics-certs\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.616528 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.616498    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.616696 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.616538    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vpv6\" (UniqueName: \"kubernetes.io/projected/0e985862-f6c6-4a53-b85a-a395b3177e04-kube-api-access-4vpv6\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.616696 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.616564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e985862-f6c6-4a53-b85a-a395b3177e04-openshift-service-ca\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.616696 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.616637    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-metrics-certs\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.616696 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:24.616673    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-serving-cert: secret "maas-api-e2e-ait-169213ba-serving-cert" not found
Sep 10 21:13:24.616934 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:24.616750    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls podName:0e985862-f6c6-4a53-b85a-a395b3177e04 nodeName:}" failed. No retries permitted until 2026-09-10 21:13:25.116729888 +0000 UTC m=+892.118976223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls") pod "maas-api-e2e-ait-169213ba-7997766cf9-jw7vk" (UID: "0e985862-f6c6-4a53-b85a-a395b3177e04") : secret "maas-api-e2e-ait-169213ba-serving-cert" not found
Sep 10 21:13:24.617276 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.617251    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e985862-f6c6-4a53-b85a-a395b3177e04-openshift-service-ca\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.619175 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.619155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-metrics-certs\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:24.627770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:24.627752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vpv6\" (UniqueName: \"kubernetes.io/projected/0e985862-f6c6-4a53-b85a-a395b3177e04-kube-api-access-4vpv6\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:25.121574 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:25.121535    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:25.124145 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:25.124103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls\") pod \"maas-api-e2e-ait-169213ba-7997766cf9-jw7vk\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:25.361991 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:25.361950    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:25.640114 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:25.640086    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"]
Sep 10 21:13:25.641368 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:13:25.641339    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e985862_f6c6_4a53_b85a_a395b3177e04.slice/crio-cfb3ff80db2abedfea40451fe4390f6c34fc006c0c67640380f6c1e7e7e52169 WatchSource:0}: Error finding container cfb3ff80db2abedfea40451fe4390f6c34fc006c0c67640380f6c1e7e7e52169: Status 404 returned error can't find the container with id cfb3ff80db2abedfea40451fe4390f6c34fc006c0c67640380f6c1e7e7e52169
Sep 10 21:13:25.990371 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:25.990255    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk" event={"ID":"0e985862-f6c6-4a53-b85a-a395b3177e04","Type":"ContainerStarted","Data":"cfb3ff80db2abedfea40451fe4390f6c34fc006c0c67640380f6c1e7e7e52169"}
Sep 10 21:13:26.997959 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:26.997920    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk" event={"ID":"0e985862-f6c6-4a53-b85a-a395b3177e04","Type":"ContainerStarted","Data":"7ed57d7c01a904d4b3b2e20cd3068991a16570b28238622db08d3bc5a5850d21"}
Sep 10 21:13:26.998380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:26.997981    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:27.016889 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:27.016840    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk" podStartSLOduration=2.537564895 podStartE2EDuration="3.016825s" totalImagesPullingTime="479.260105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:27.014249199 +0000 UTC m=+894.016495717" watchObservedRunningTime="2026-09-10 21:13:27.016825 +0000 UTC m=+894.019071348"
Sep 10 21:13:33.009473 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:33.009435    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:44.102767 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:44.102733    2580 generic.go:356] "Generic (PLEG): container finished" podID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerID="79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f" exitCode=0
Sep 10 21:13:44.103311 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:44.102818    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" event={"ID":"34aaaf9a-378f-4ce0-9936-d53fd9e4b099","Type":"ContainerDied","Data":"79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f"}
Sep 10 21:13:44.103385 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:44.103348    2580 scope.go:117] "RemoveContainer" containerID="79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f"
Sep 10 21:13:44.104432 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:44.104404    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:44.104547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:44.104488    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:44.104613 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:44.104546    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:45.026304 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.026267    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"]
Sep 10 21:13:45.026574 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.026541    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v" podUID="4925272e-744b-4ff9-a0ee-3571615d3b69" containerName="maas-api" containerID="cri-o://73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456" gracePeriod=30
Sep 10 21:13:45.115144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.111833    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"]
Sep 10 21:13:45.115144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.112138    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z" podUID="ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c" containerName="payload-pre-processing" containerID="cri-o://97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af" gracePeriod=30
Sep 10 21:13:45.129975 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.129312    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" event={"ID":"34aaaf9a-378f-4ce0-9936-d53fd9e4b099","Type":"ContainerStarted","Data":"a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7"}
Sep 10 21:13:45.419951 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.419797    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:13:45.423628 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.423596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4925272e-744b-4ff9-a0ee-3571615d3b69-openshift-service-ca\") pod \"4925272e-744b-4ff9-a0ee-3571615d3b69\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") "
Sep 10 21:13:45.423753 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.423666    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-metrics-certs\") pod \"4925272e-744b-4ff9-a0ee-3571615d3b69\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") "
Sep 10 21:13:45.423753 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.423724    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8z5bh\" (UniqueName: \"kubernetes.io/projected/4925272e-744b-4ff9-a0ee-3571615d3b69-kube-api-access-8z5bh\") pod \"4925272e-744b-4ff9-a0ee-3571615d3b69\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") "
Sep 10 21:13:45.424090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.423877    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4925272e-744b-4ff9-a0ee-3571615d3b69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4925272e-744b-4ff9-a0ee-3571615d3b69" (UID: "4925272e-744b-4ff9-a0ee-3571615d3b69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:45.424090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.423948    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4925272e-744b-4ff9-a0ee-3571615d3b69-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:45.425988 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.425959    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4925272e-744b-4ff9-a0ee-3571615d3b69" (UID: "4925272e-744b-4ff9-a0ee-3571615d3b69"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:45.426085 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.426066    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4925272e-744b-4ff9-a0ee-3571615d3b69-kube-api-access-8z5bh" (OuterVolumeSpecName: "kube-api-access-8z5bh") pod "4925272e-744b-4ff9-a0ee-3571615d3b69" (UID: "4925272e-744b-4ff9-a0ee-3571615d3b69"). InnerVolumeSpecName "kube-api-access-8z5bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:45.462814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.462790    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:13:45.524359 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.524332    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-maas-api-tls\") pod \"4925272e-744b-4ff9-a0ee-3571615d3b69\" (UID: \"4925272e-744b-4ff9-a0ee-3571615d3b69\") "
Sep 10 21:13:45.524735 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.524712    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:45.524735 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.524734    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z5bh\" (UniqueName: \"kubernetes.io/projected/4925272e-744b-4ff9-a0ee-3571615d3b69-kube-api-access-8z5bh\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:45.526481 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.526459    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4925272e-744b-4ff9-a0ee-3571615d3b69" (UID: "4925272e-744b-4ff9-a0ee-3571615d3b69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:45.625886 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.625859    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-plugins-config-volume\") pod \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\" (UID: \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\") "
Sep 10 21:13:45.626080 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.625948    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5z8p\" (UniqueName: \"kubernetes.io/projected/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-kube-api-access-r5z8p\") pod \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\" (UID: \"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c\") "
Sep 10 21:13:45.626185 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.626161    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4925272e-744b-4ff9-a0ee-3571615d3b69-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:45.626332 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.626307    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c" (UID: "ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:45.628106 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.628083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-kube-api-access-r5z8p" (OuterVolumeSpecName: "kube-api-access-r5z8p") pod "ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c" (UID: "ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c"). InnerVolumeSpecName "kube-api-access-r5z8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:45.726634 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.726597    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:45.726634 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.726629    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5z8p\" (UniqueName: \"kubernetes.io/projected/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c-kube-api-access-r5z8p\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:45.996995 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.996898    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:45.997180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.997017    2580 patch_prober.go:28] interesting pod/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 21:13:45.997180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:45.997058    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 21:13:46.137002 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.136969    2580 generic.go:356] "Generic (PLEG): container finished" podID="4925272e-744b-4ff9-a0ee-3571615d3b69" containerID="73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456" exitCode=0
Sep 10 21:13:46.137437 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.137017    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"
Sep 10 21:13:46.137437 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.137011    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v" event={"ID":"4925272e-744b-4ff9-a0ee-3571615d3b69","Type":"ContainerDied","Data":"73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456"}
Sep 10 21:13:46.137437 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.137166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v" event={"ID":"4925272e-744b-4ff9-a0ee-3571615d3b69","Type":"ContainerDied","Data":"0628152af193f5d91bf90ea65ee6aa6e2dc7a497e7fda5fb913a3ed5630413d3"}
Sep 10 21:13:46.137437 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.137193    2580 scope.go:117] "RemoveContainer" containerID="73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456"
Sep 10 21:13:46.138619 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.138572    2580 generic.go:356] "Generic (PLEG): container finished" podID="ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c" containerID="97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af" exitCode=0
Sep 10 21:13:46.138678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.138644    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"
Sep 10 21:13:46.138678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.138656    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z" event={"ID":"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c","Type":"ContainerDied","Data":"97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af"}
Sep 10 21:13:46.138758 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.138713    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z" event={"ID":"ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c","Type":"ContainerDied","Data":"24f739ee96f26c9050ad5dc8ec7ebe26b6d5252c82aa0525d4b747e84f232503"}
Sep 10 21:13:46.148765 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.148744    2580 scope.go:117] "RemoveContainer" containerID="73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456"
Sep 10 21:13:46.149073 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:46.149055    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456\": container with ID starting with 73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456 not found: ID does not exist" containerID="73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456"
Sep 10 21:13:46.149194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.149082    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456"} err="failed to get container status \"73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456\": rpc error: code = NotFound desc = could not find container \"73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456\": container with ID starting with 73e3366d089ebf9de1d3b9b5afb33bff2d474169737ddc9b18a90af7e0553456 not found: ID does not exist"
Sep 10 21:13:46.149194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.149099    2580 scope.go:117] "RemoveContainer" containerID="97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af"
Sep 10 21:13:46.160101 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.160077    2580 scope.go:117] "RemoveContainer" containerID="97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af"
Sep 10 21:13:46.160426 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:46.160400    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af\": container with ID starting with 97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af not found: ID does not exist" containerID="97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af"
Sep 10 21:13:46.160496 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.160433    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af"} err="failed to get container status \"97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af\": rpc error: code = NotFound desc = could not find container \"97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af\": container with ID starting with 97cfa8a9be6e39aac474e686feb339c14081927012aa31dc243ce455c76307af not found: ID does not exist"
Sep 10 21:13:46.183158 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.178002    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"]
Sep 10 21:13:46.190224 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.190191    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-5e2fa5-78fb855765-5lr2v"]
Sep 10 21:13:46.203246 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.203214    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"]
Sep 10 21:13:46.208816 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.208791    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5e2fa5-7f84b8fb6b-4ld9z"]
Sep 10 21:13:46.997205 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.997151    2580 patch_prober.go:28] interesting pod/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 21:13:46.997410 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:46.997217    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 21:13:47.519811 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:47.519772    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4925272e-744b-4ff9-a0ee-3571615d3b69" path="/var/lib/kubelet/pods/4925272e-744b-4ff9-a0ee-3571615d3b69/volumes"
Sep 10 21:13:47.520329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:47.520308    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c" path="/var/lib/kubelet/pods/ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c/volumes"
Sep 10 21:13:47.997524 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:47.997484    2580 patch_prober.go:28] interesting pod/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 21:13:47.997846 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:47.997810    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 21:13:48.997791 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:48.997753    2580 patch_prober.go:28] interesting pod/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 21:13:48.998394 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:48.998360    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 21:13:49.997466 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:49.997423    2580 patch_prober.go:28] interesting pod/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused" start-of-body=
Sep 10 21:13:49.997713 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:49.997499    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.61:15021/healthz/ready\": dial tcp 10.133.0.61:15021: connect: connection refused"
Sep 10 21:13:51.001516 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:51.001484    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:51.001892 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:51.001753    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:51.002392 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:51.002373    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:13:52.518783 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:52.518748    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"]
Sep 10 21:13:52.519246 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:52.518950    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy" containerID="cri-o://5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af" gracePeriod=30
Sep 10 21:13:55.047642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.047611    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"]
Sep 10 21:13:55.048049 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.047831    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk" podUID="0e985862-f6c6-4a53-b85a-a395b3177e04" containerName="maas-api" containerID="cri-o://7ed57d7c01a904d4b3b2e20cd3068991a16570b28238622db08d3bc5a5850d21" gracePeriod=30
Sep 10 21:13:55.204620 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.204587    2580 generic.go:356] "Generic (PLEG): container finished" podID="0e985862-f6c6-4a53-b85a-a395b3177e04" containerID="7ed57d7c01a904d4b3b2e20cd3068991a16570b28238622db08d3bc5a5850d21" exitCode=0
Sep 10 21:13:55.204788 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.204645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk" event={"ID":"0e985862-f6c6-4a53-b85a-a395b3177e04","Type":"ContainerDied","Data":"7ed57d7c01a904d4b3b2e20cd3068991a16570b28238622db08d3bc5a5850d21"}
Sep 10 21:13:55.310942 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.310882    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:55.405370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.405335    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e985862-f6c6-4a53-b85a-a395b3177e04-openshift-service-ca\") pod \"0e985862-f6c6-4a53-b85a-a395b3177e04\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") "
Sep 10 21:13:55.405535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.405384    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vpv6\" (UniqueName: \"kubernetes.io/projected/0e985862-f6c6-4a53-b85a-a395b3177e04-kube-api-access-4vpv6\") pod \"0e985862-f6c6-4a53-b85a-a395b3177e04\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") "
Sep 10 21:13:55.405535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.405428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-metrics-certs\") pod \"0e985862-f6c6-4a53-b85a-a395b3177e04\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") "
Sep 10 21:13:55.405535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.405483    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls\") pod \"0e985862-f6c6-4a53-b85a-a395b3177e04\" (UID: \"0e985862-f6c6-4a53-b85a-a395b3177e04\") "
Sep 10 21:13:55.405766 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.405737    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e985862-f6c6-4a53-b85a-a395b3177e04-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e985862-f6c6-4a53-b85a-a395b3177e04" (UID: "0e985862-f6c6-4a53-b85a-a395b3177e04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:55.407728 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.407696    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0e985862-f6c6-4a53-b85a-a395b3177e04" (UID: "0e985862-f6c6-4a53-b85a-a395b3177e04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:55.407870 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.407733    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e985862-f6c6-4a53-b85a-a395b3177e04-kube-api-access-4vpv6" (OuterVolumeSpecName: "kube-api-access-4vpv6") pod "0e985862-f6c6-4a53-b85a-a395b3177e04" (UID: "0e985862-f6c6-4a53-b85a-a395b3177e04"). InnerVolumeSpecName "kube-api-access-4vpv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:55.407870 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.407788    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e985862-f6c6-4a53-b85a-a395b3177e04" (UID: "0e985862-f6c6-4a53-b85a-a395b3177e04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:13:55.506440 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.506396    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:55.506440 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.506431    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e985862-f6c6-4a53-b85a-a395b3177e04-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:55.506440 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.506442    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e985862-f6c6-4a53-b85a-a395b3177e04-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:55.506690 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:55.506451    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vpv6\" (UniqueName: \"kubernetes.io/projected/0e985862-f6c6-4a53-b85a-a395b3177e04-kube-api-access-4vpv6\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:56.211443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:56.211410    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk" event={"ID":"0e985862-f6c6-4a53-b85a-a395b3177e04","Type":"ContainerDied","Data":"cfb3ff80db2abedfea40451fe4390f6c34fc006c0c67640380f6c1e7e7e52169"}
Sep 10 21:13:56.211930 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:56.211455    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"
Sep 10 21:13:56.211930 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:56.211473    2580 scope.go:117] "RemoveContainer" containerID="7ed57d7c01a904d4b3b2e20cd3068991a16570b28238622db08d3bc5a5850d21"
Sep 10 21:13:56.228602 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:56.228572    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"]
Sep 10 21:13:56.231139 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:56.231096    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-169213ba-7997766cf9-jw7vk"]
Sep 10 21:13:56.465192 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:56.465101    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"]
Sep 10 21:13:56.465341 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:56.465310    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy" containerID="cri-o://a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7" gracePeriod=30
Sep 10 21:13:57.223856 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.223826    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"]
Sep 10 21:13:57.224286 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224271    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c" containerName="payload-pre-processing"
Sep 10 21:13:57.224356 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224290    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c" containerName="payload-pre-processing"
Sep 10 21:13:57.224356 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224304    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e985862-f6c6-4a53-b85a-a395b3177e04" containerName="maas-api"
Sep 10 21:13:57.224356 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224309    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e985862-f6c6-4a53-b85a-a395b3177e04" containerName="maas-api"
Sep 10 21:13:57.224356 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224319    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4925272e-744b-4ff9-a0ee-3571615d3b69" containerName="maas-api"
Sep 10 21:13:57.224356 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224325    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4925272e-744b-4ff9-a0ee-3571615d3b69" containerName="maas-api"
Sep 10 21:13:57.224557 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224393    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4925272e-744b-4ff9-a0ee-3571615d3b69" containerName="maas-api"
Sep 10 21:13:57.224557 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224402    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff909dd7-d6a4-4687-8a8b-2d7783dd1c5c" containerName="payload-pre-processing"
Sep 10 21:13:57.224557 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.224411    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e985862-f6c6-4a53-b85a-a395b3177e04" containerName="maas-api"
Sep 10 21:13:57.227316 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.227296    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.229768 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.229733    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-3002a381-openshift-default-dockercfg-6khwt\""
Sep 10 21:13:57.238337 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.238317    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"]
Sep 10 21:13:57.321550 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321518    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-credential-socket\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.321550 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-certs\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.321814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321582    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-data\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.321814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399c36f5-82d2-4c10-ab6a-604cb3084d88-istiod-ca-cert\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.321814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321721    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-envoy\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.321814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321780    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-socket\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.321814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6kfl\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-kube-api-access-j6kfl\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.322023 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321849    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-token\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.322023 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.321932    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-podinfo\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.422888 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.422853    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-credential-socket\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.422888 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.422890    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-certs\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.422924    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-data\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.422948    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399c36f5-82d2-4c10-ab6a-604cb3084d88-istiod-ca-cert\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.422970    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-envoy\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.423001    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-socket\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.423020    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6kfl\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-kube-api-access-j6kfl\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.423043    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-token\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.423316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-podinfo\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.423350    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-data\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.423388    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-credential-socket\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.423431    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-certs\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.423646 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.423545    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-socket\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.424026 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.424003    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399c36f5-82d2-4c10-ab6a-604cb3084d88-istiod-ca-cert\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.425336 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.425318    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-envoy\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.425795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.425774    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-podinfo\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.431595 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.431573    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-token\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.431859 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.431839    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6kfl\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-kube-api-access-j6kfl\") pod \"e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") " pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.519711 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.519641    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e985862-f6c6-4a53-b85a-a395b3177e04" path="/var/lib/kubelet/pods/0e985862-f6c6-4a53-b85a-a395b3177e04/volumes"
Sep 10 21:13:57.539004 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.538976    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:57.785105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.785045    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:13:57.816705 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.816673    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"]
Sep 10 21:13:57.817923 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:13:57.817899    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod399c36f5_82d2_4c10_ab6a_604cb3084d88.slice/crio-11c4b2974edf870742ed01a5e8bc3c3dc34747c21382326fe245dd5be0cfb839 WatchSource:0}: Error finding container 11c4b2974edf870742ed01a5e8bc3c3dc34747c21382326fe245dd5be0cfb839: Status 404 returned error can't find the container with id 11c4b2974edf870742ed01a5e8bc3c3dc34747c21382326fe245dd5be0cfb839
Sep 10 21:13:57.820650 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.820622    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:57.820714 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.820682    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:57.820714 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.820711    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:13:57.828133 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828095    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-socket\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828247 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828178    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-credential-socket\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828247 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828206    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bzzw\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-kube-api-access-5bzzw\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828359 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828247    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-token\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828359 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828270    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-certs\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828863 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828413    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-envoy\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828863 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828480    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc529b33-6703-43ba-8ea6-e58edd0c7689-istiod-ca-cert\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828996 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-data\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828996 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828938    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-podinfo\") pod \"fc529b33-6703-43ba-8ea6-e58edd0c7689\" (UID: \"fc529b33-6703-43ba-8ea6-e58edd0c7689\") "
Sep 10 21:13:57.828996 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.828961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:57.829176 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.829047    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:57.829594 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.829413    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:57.829594 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.829566    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:57.829594 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.829467    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:57.829594 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.829494    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc529b33-6703-43ba-8ea6-e58edd0c7689-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:13:57.829887 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.829662    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:57.831543 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.831512    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:57.831543 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.831519    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:13:57.831681 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.831583    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-kube-api-access-5bzzw" (OuterVolumeSpecName: "kube-api-access-5bzzw") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "kube-api-access-5bzzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:13:57.831681 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.831644    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc529b33-6703-43ba-8ea6-e58edd0c7689" (UID: "fc529b33-6703-43ba-8ea6-e58edd0c7689"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:13:57.930947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.930920    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:57.930947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.930942    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:57.930947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.930950    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc529b33-6703-43ba-8ea6-e58edd0c7689-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:57.931190 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.930959    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:57.931190 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.930968    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc529b33-6703-43ba-8ea6-e58edd0c7689-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:57.931190 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.930975    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc529b33-6703-43ba-8ea6-e58edd0c7689-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:57.931190 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:57.930988    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bzzw\" (UniqueName: \"kubernetes.io/projected/fc529b33-6703-43ba-8ea6-e58edd0c7689-kube-api-access-5bzzw\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:13:58.226131 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.226084    2580 generic.go:356] "Generic (PLEG): container finished" podID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerID="5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af" exitCode=0
Sep 10 21:13:58.226679 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.226158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" event={"ID":"fc529b33-6703-43ba-8ea6-e58edd0c7689","Type":"ContainerDied","Data":"5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af"}
Sep 10 21:13:58.226679 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.226182    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"
Sep 10 21:13:58.226679 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.226218    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt" event={"ID":"fc529b33-6703-43ba-8ea6-e58edd0c7689","Type":"ContainerDied","Data":"90d20b6f56d791e867c87a823d989a62a14225187114c306d7e47fafe7ecd983"}
Sep 10 21:13:58.226679 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.226243    2580 scope.go:117] "RemoveContainer" containerID="5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af"
Sep 10 21:13:58.228314 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.228292    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" event={"ID":"399c36f5-82d2-4c10-ab6a-604cb3084d88","Type":"ContainerStarted","Data":"02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887"}
Sep 10 21:13:58.228456 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.228344    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" event={"ID":"399c36f5-82d2-4c10-ab6a-604cb3084d88","Type":"ContainerStarted","Data":"11c4b2974edf870742ed01a5e8bc3c3dc34747c21382326fe245dd5be0cfb839"}
Sep 10 21:13:58.237368 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.237353    2580 scope.go:117] "RemoveContainer" containerID="a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476"
Sep 10 21:13:58.247189 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.247152    2580 scope.go:117] "RemoveContainer" containerID="5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af"
Sep 10 21:13:58.247759 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:58.247716    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af\": container with ID starting with 5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af not found: ID does not exist" containerID="5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af"
Sep 10 21:13:58.247932 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.247805    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af"} err="failed to get container status \"5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af\": rpc error: code = NotFound desc = could not find container \"5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af\": container with ID starting with 5cba0461c533b2fe7d2a8b70d8fb45087aa2367eadcc1445a20ba12b857146af not found: ID does not exist"
Sep 10 21:13:58.247932 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.247839    2580 scope.go:117] "RemoveContainer" containerID="a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476"
Sep 10 21:13:58.247932 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.247817    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podStartSLOduration=1.247798901 podStartE2EDuration="1.247798901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:13:58.247381426 +0000 UTC m=+925.249627790" watchObservedRunningTime="2026-09-10 21:13:58.247798901 +0000 UTC m=+925.250045249"
Sep 10 21:13:58.248374 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:58.248244    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476\": container with ID starting with a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476 not found: ID does not exist" containerID="a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476"
Sep 10 21:13:58.248374 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.248286    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476"} err="failed to get container status \"a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476\": rpc error: code = NotFound desc = could not find container \"a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476\": container with ID starting with a089fac7b25c57bbdeb066a5627b21a59cf3025e10bb444eed564bf603343476 not found: ID does not exist"
Sep 10 21:13:58.261237 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.261216    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"]
Sep 10 21:13:58.266718 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.266699    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-5e2fa5-openshift-default-7b5cb484d6-b4gtt"]
Sep 10 21:13:58.539278 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.539196    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:58.544756 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.544730    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:58.591090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.591061    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"]
Sep 10 21:13:58.591602 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.591579    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy"
Sep 10 21:13:58.591602 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.591600    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy"
Sep 10 21:13:58.591770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.591618    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy"
Sep 10 21:13:58.591770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.591627    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy"
Sep 10 21:13:58.591770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.591708    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy"
Sep 10 21:13:58.591770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.591715    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" containerName="istio-proxy"
Sep 10 21:13:58.594017 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.593895    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.596405 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.596386    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-2bbc78-serving-cert\""
Sep 10 21:13:58.607344 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.607320    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"]
Sep 10 21:13:58.637772 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.637745    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.637903 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.637797    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a5b171-e02a-4863-8740-9acd5c6b2d44-openshift-service-ca\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.637903 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.637836    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-metrics-certs\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.637903 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.637878    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dp85j\" (UniqueName: \"kubernetes.io/projected/80a5b171-e02a-4863-8740-9acd5c6b2d44-kube-api-access-dp85j\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.738731 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.738697    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dp85j\" (UniqueName: \"kubernetes.io/projected/80a5b171-e02a-4863-8740-9acd5c6b2d44-kube-api-access-dp85j\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.738876 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.738758    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.738876 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.738806    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a5b171-e02a-4863-8740-9acd5c6b2d44-openshift-service-ca\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.738876 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.738827    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-metrics-certs\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.739070 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:58.738987    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-serving-cert: secret "maas-api-e2e-explicit-2bbc78-serving-cert" not found
Sep 10 21:13:58.739179 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:13:58.739106    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls podName:80a5b171-e02a-4863-8740-9acd5c6b2d44 nodeName:}" failed. No retries permitted until 2026-09-10 21:13:59.239044651 +0000 UTC m=+926.241290977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls") pod "maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4" (UID: "80a5b171-e02a-4863-8740-9acd5c6b2d44") : secret "maas-api-e2e-explicit-2bbc78-serving-cert" not found
Sep 10 21:13:58.739958 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.739930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a5b171-e02a-4863-8740-9acd5c6b2d44-openshift-service-ca\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.741656 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.741633    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-metrics-certs\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:58.749313 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:58.749291    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dp85j\" (UniqueName: \"kubernetes.io/projected/80a5b171-e02a-4863-8740-9acd5c6b2d44-kube-api-access-dp85j\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:59.235000 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.234966    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:59.235951 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.235902    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:13:59.242138 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.242102    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:59.244595 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.244566    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls\") pod \"maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:59.291437 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.291399    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"]
Sep 10 21:13:59.294829 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.294806    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:13:59.296967 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.296945    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-2bbc78\""
Sep 10 21:13:59.297070 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.297033    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-2bbc78-dockercfg-4zd7b\""
Sep 10 21:13:59.301839 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.301815    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"]
Sep 10 21:13:59.343484 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.343455    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f664264-24c5-4e6c-b9b5-107cabf5f812-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb\" (UID: \"9f664264-24c5-4e6c-b9b5-107cabf5f812\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:13:59.343699 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.343670    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6fhv\" (UniqueName: \"kubernetes.io/projected/9f664264-24c5-4e6c-b9b5-107cabf5f812-kube-api-access-v6fhv\") pod \"payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb\" (UID: \"9f664264-24c5-4e6c-b9b5-107cabf5f812\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:13:59.444884 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.444847    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f664264-24c5-4e6c-b9b5-107cabf5f812-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb\" (UID: \"9f664264-24c5-4e6c-b9b5-107cabf5f812\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:13:59.444884 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.444886    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6fhv\" (UniqueName: \"kubernetes.io/projected/9f664264-24c5-4e6c-b9b5-107cabf5f812-kube-api-access-v6fhv\") pod \"payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb\" (UID: \"9f664264-24c5-4e6c-b9b5-107cabf5f812\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:13:59.445722 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.445695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f664264-24c5-4e6c-b9b5-107cabf5f812-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb\" (UID: \"9f664264-24c5-4e6c-b9b5-107cabf5f812\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:13:59.453920 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.453896    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6fhv\" (UniqueName: \"kubernetes.io/projected/9f664264-24c5-4e6c-b9b5-107cabf5f812-kube-api-access-v6fhv\") pod \"payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb\" (UID: \"9f664264-24c5-4e6c-b9b5-107cabf5f812\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:13:59.508493 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.508403    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:13:59.521898 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.521864    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc529b33-6703-43ba-8ea6-e58edd0c7689" path="/var/lib/kubelet/pods/fc529b33-6703-43ba-8ea6-e58edd0c7689/volumes"
Sep 10 21:13:59.608273 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.608248    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:13:59.782913 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.782885    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"]
Sep 10 21:13:59.784780 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:13:59.784750    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a5b171_e02a_4863_8740_9acd5c6b2d44.slice/crio-27c6b61b9a25236f25365defa4088444f69bdbe9b38124245131a662cb870a3c WatchSource:0}: Error finding container 27c6b61b9a25236f25365defa4088444f69bdbe9b38124245131a662cb870a3c: Status 404 returned error can't find the container with id 27c6b61b9a25236f25365defa4088444f69bdbe9b38124245131a662cb870a3c
Sep 10 21:13:59.786217 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.786198    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:13:59.885515 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:13:59.885493    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"]
Sep 10 21:13:59.887231 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:13:59.887204    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f664264_24c5_4e6c_b9b5_107cabf5f812.slice/crio-8f81ead44fb052a371a0819aa2b5d70f47df67b96fa6271cc344e857c6457994 WatchSource:0}: Error finding container 8f81ead44fb052a371a0819aa2b5d70f47df67b96fa6271cc344e857c6457994: Status 404 returned error can't find the container with id 8f81ead44fb052a371a0819aa2b5d70f47df67b96fa6271cc344e857c6457994
Sep 10 21:14:00.241721 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.241686    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4" event={"ID":"80a5b171-e02a-4863-8740-9acd5c6b2d44","Type":"ContainerStarted","Data":"27c6b61b9a25236f25365defa4088444f69bdbe9b38124245131a662cb870a3c"}
Sep 10 21:14:00.243357 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.243331    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb" event={"ID":"9f664264-24c5-4e6c-b9b5-107cabf5f812","Type":"ContainerStarted","Data":"ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87"}
Sep 10 21:14:00.243487 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.243377    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb" event={"ID":"9f664264-24c5-4e6c-b9b5-107cabf5f812","Type":"ContainerStarted","Data":"8f81ead44fb052a371a0819aa2b5d70f47df67b96fa6271cc344e857c6457994"}
Sep 10 21:14:00.243716 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.243696    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:14:00.261512 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.261462    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb" podStartSLOduration=1.261448589 podStartE2EDuration="1.261448589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:00.259770222 +0000 UTC m=+927.262016569" watchObservedRunningTime="2026-09-10 21:14:00.261448589 +0000 UTC m=+927.263694935"
Sep 10 21:14:00.832840 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.832802    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"]
Sep 10 21:14:00.835365 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.835346    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.837895 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.837855    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-3002a381-serving-cert\""
Sep 10 21:14:00.848108 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.847609    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"]
Sep 10 21:14:00.857625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.857597    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9f257-4081-4e30-9005-74f29e2f87d3-openshift-service-ca\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.857750 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.857645    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw65v\" (UniqueName: \"kubernetes.io/projected/cfa9f257-4081-4e30-9005-74f29e2f87d3-kube-api-access-tw65v\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.857815 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.857748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.857815 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.857777    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-metrics-certs\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.958547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.958513    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9f257-4081-4e30-9005-74f29e2f87d3-openshift-service-ca\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.958547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.958551    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw65v\" (UniqueName: \"kubernetes.io/projected/cfa9f257-4081-4e30-9005-74f29e2f87d3-kube-api-access-tw65v\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.958770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.958599    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.958770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.958625    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-metrics-certs\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.958770 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:14:00.958752    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-serving-cert: secret "maas-api-e2e-derive-3002a381-serving-cert" not found
Sep 10 21:14:00.958882 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:14:00.958814    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls podName:cfa9f257-4081-4e30-9005-74f29e2f87d3 nodeName:}" failed. No retries permitted until 2026-09-10 21:14:01.458796652 +0000 UTC m=+928.461042992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls") pod "maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r" (UID: "cfa9f257-4081-4e30-9005-74f29e2f87d3") : secret "maas-api-e2e-derive-3002a381-serving-cert" not found
Sep 10 21:14:00.959193 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.959167    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9f257-4081-4e30-9005-74f29e2f87d3-openshift-service-ca\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.961107 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.961087    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-metrics-certs\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:00.968351 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:00.968333    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw65v\" (UniqueName: \"kubernetes.io/projected/cfa9f257-4081-4e30-9005-74f29e2f87d3-kube-api-access-tw65v\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:01.253091 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.253048    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4" event={"ID":"80a5b171-e02a-4863-8740-9acd5c6b2d44","Type":"ContainerStarted","Data":"1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a"}
Sep 10 21:14:01.253539 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.253168    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:14:01.273042 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.272995    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4" podStartSLOduration=2.718479531 podStartE2EDuration="3.272980226s" totalImagesPullingTime="554.500695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:13:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:01.269945633 +0000 UTC m=+928.272191980" watchObservedRunningTime="2026-09-10 21:14:01.272980226 +0000 UTC m=+928.275226610"
Sep 10 21:14:01.462856 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.462817    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:01.465578 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.465552    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls\") pod \"maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:01.721159 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.721135    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:14:01.751285 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.751256    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:01.765366 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765339    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-certs\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765501 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765380    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfpwb\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-kube-api-access-xfpwb\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765501 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765428    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-token\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765501 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765457    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-socket\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765501 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765496    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istiod-ca-cert\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765722 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-podinfo\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765722 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765592    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-envoy\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765722 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765632    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-credential-socket\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765722 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765626    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:01.765722 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765654    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-data\") pod \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\" (UID: \"34aaaf9a-378f-4ce0-9936-d53fd9e4b099\") "
Sep 10 21:14:01.765982 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.765926    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.766108 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.766057    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:01.767106 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.767077    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-data" (OuterVolumeSpecName: "istio-data") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:01.767424 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.767399    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:01.767500 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.767477    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:01.768483 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.768455    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:01.768579 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.768500    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:14:01.768644 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.768599    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-token" (OuterVolumeSpecName: "istio-token") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:01.768865 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.768843    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-kube-api-access-xfpwb" (OuterVolumeSpecName: "kube-api-access-xfpwb") pod "34aaaf9a-378f-4ce0-9936-d53fd9e4b099" (UID: "34aaaf9a-378f-4ce0-9936-d53fd9e4b099"). InnerVolumeSpecName "kube-api-access-xfpwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:01.866663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.866595    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.866663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.866620    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.866663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.866629    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.866663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.866638    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.866663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.866646    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.866663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.866655    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.866663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.866663    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:01.866968 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:01.866672    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfpwb\" (UniqueName: \"kubernetes.io/projected/34aaaf9a-378f-4ce0-9936-d53fd9e4b099-kube-api-access-xfpwb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:02.028369 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.028344    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"]
Sep 10 21:14:02.030342 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:14:02.030317    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa9f257_4081_4e30_9005_74f29e2f87d3.slice/crio-211f1a096a030f53ca3ef818d5c6ee19cdd1e7d2677887faf71e7dcc45b8acff WatchSource:0}: Error finding container 211f1a096a030f53ca3ef818d5c6ee19cdd1e7d2677887faf71e7dcc45b8acff: Status 404 returned error can't find the container with id 211f1a096a030f53ca3ef818d5c6ee19cdd1e7d2677887faf71e7dcc45b8acff
Sep 10 21:14:02.260616 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.260579    2580 generic.go:356] "Generic (PLEG): container finished" podID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerID="a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7" exitCode=0
Sep 10 21:14:02.261055 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.260634    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"
Sep 10 21:14:02.261055 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.260666    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" event={"ID":"34aaaf9a-378f-4ce0-9936-d53fd9e4b099","Type":"ContainerDied","Data":"a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7"}
Sep 10 21:14:02.261055 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.260741    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p" event={"ID":"34aaaf9a-378f-4ce0-9936-d53fd9e4b099","Type":"ContainerDied","Data":"61066c2940f30f56c6bf88288dd711dfaa0ab4b1e3ecb0ee9b53a865fc15f0dc"}
Sep 10 21:14:02.261055 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.260775    2580 scope.go:117] "RemoveContainer" containerID="a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7"
Sep 10 21:14:02.262057 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.262032    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r" event={"ID":"cfa9f257-4081-4e30-9005-74f29e2f87d3","Type":"ContainerStarted","Data":"211f1a096a030f53ca3ef818d5c6ee19cdd1e7d2677887faf71e7dcc45b8acff"}
Sep 10 21:14:02.271115 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.271093    2580 scope.go:117] "RemoveContainer" containerID="79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f"
Sep 10 21:14:02.279761 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.279741    2580 scope.go:117] "RemoveContainer" containerID="a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7"
Sep 10 21:14:02.280022 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:14:02.280003    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7\": container with ID starting with a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7 not found: ID does not exist" containerID="a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7"
Sep 10 21:14:02.280072 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.280030    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7"} err="failed to get container status \"a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7\": rpc error: code = NotFound desc = could not find container \"a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7\": container with ID starting with a68e1c5351eed635c3660154075782faa6c08e8b9279df2e9406cffc0bcd13b7 not found: ID does not exist"
Sep 10 21:14:02.280072 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.280046    2580 scope.go:117] "RemoveContainer" containerID="79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f"
Sep 10 21:14:02.280419 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:14:02.280401    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f\": container with ID starting with 79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f not found: ID does not exist" containerID="79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f"
Sep 10 21:14:02.280465 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.280426    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f"} err="failed to get container status \"79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f\": rpc error: code = NotFound desc = could not find container \"79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f\": container with ID starting with 79ac5b4579f0c22698f29e18902b2e74f0dc6a7247bda679cb7a0eb92d5e6f0f not found: ID does not exist"
Sep 10 21:14:02.285614 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.285593    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"]
Sep 10 21:14:02.289151 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:02.289114    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-169213ba-openshift-default-7c467c95d-j4d9p"]
Sep 10 21:14:03.269781 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:03.269746    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r" event={"ID":"cfa9f257-4081-4e30-9005-74f29e2f87d3","Type":"ContainerStarted","Data":"a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0"}
Sep 10 21:14:03.270205 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:03.269875    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:03.288018 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:03.287971    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r" podStartSLOduration=2.920430446 podStartE2EDuration="3.287953886s" totalImagesPullingTime="367.52344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:14:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:14:03.286841517 +0000 UTC m=+930.289087865" watchObservedRunningTime="2026-09-10 21:14:03.287953886 +0000 UTC m=+930.290200257"
Sep 10 21:14:03.519888 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:03.519811    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" path="/var/lib/kubelet/pods/34aaaf9a-378f-4ce0-9936-d53fd9e4b099/volumes"
Sep 10 21:14:07.268219 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:07.268183    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:14:09.294393 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:09.294353    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:11.256133 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:11.256087    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:14:14.237024 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:14.236989    2580 patch_prober.go:28] interesting pod/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 10 21:14:14.237434 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:14.237042    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 10 21:14:14.349618 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:14.349586    2580 generic.go:356] "Generic (PLEG): container finished" podID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerID="02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887" exitCode=0
Sep 10 21:14:14.349765 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:14.349660    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" event={"ID":"399c36f5-82d2-4c10-ab6a-604cb3084d88","Type":"ContainerDied","Data":"02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887"}
Sep 10 21:14:14.350065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:14.350052    2580 scope.go:117] "RemoveContainer" containerID="02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887"
Sep 10 21:14:14.351173 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:14.351146    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:14:14.351228 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:14.351216    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:14:14.351268 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:14.351252    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:14:15.357683 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:15.357647    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" event={"ID":"399c36f5-82d2-4c10-ab6a-604cb3084d88","Type":"ContainerStarted","Data":"6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e"}
Sep 10 21:14:15.539541 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:15.539507    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:14:15.539743 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:15.539723    2580 patch_prober.go:28] interesting pod/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 10 21:14:15.539788 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:15.539770    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 10 21:14:16.539792 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:16.539751    2580 patch_prober.go:28] interesting pod/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 10 21:14:16.540203 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:16.539839    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 10 21:14:17.540223 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:17.540187    2580 patch_prober.go:28] interesting pod/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 10 21:14:17.540586 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:17.540256    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 10 21:14:18.540328 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:18.540292    2580 patch_prober.go:28] interesting pod/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 10 21:14:18.540726 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:18.540357    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 10 21:14:19.540171 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:19.540107    2580 patch_prober.go:28] interesting pod/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Sep 10 21:14:19.540361 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:19.540200    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Sep 10 21:14:20.543897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:20.543868    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:14:20.544309 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:20.544146    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:14:20.544884 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:20.544861    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:14:31.380727 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.380688    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"]
Sep 10 21:14:31.381575 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.381542    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r" podUID="cfa9f257-4081-4e30-9005-74f29e2f87d3" containerName="maas-api" containerID="cri-o://a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0" gracePeriod=30
Sep 10 21:14:31.637224 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.637163    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:31.749613 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.749580    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tw65v\" (UniqueName: \"kubernetes.io/projected/cfa9f257-4081-4e30-9005-74f29e2f87d3-kube-api-access-tw65v\") pod \"cfa9f257-4081-4e30-9005-74f29e2f87d3\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") "
Sep 10 21:14:31.749613 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.749612    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls\") pod \"cfa9f257-4081-4e30-9005-74f29e2f87d3\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") "
Sep 10 21:14:31.749812 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.749757    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9f257-4081-4e30-9005-74f29e2f87d3-openshift-service-ca\") pod \"cfa9f257-4081-4e30-9005-74f29e2f87d3\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") "
Sep 10 21:14:31.749812 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.749786    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-metrics-certs\") pod \"cfa9f257-4081-4e30-9005-74f29e2f87d3\" (UID: \"cfa9f257-4081-4e30-9005-74f29e2f87d3\") "
Sep 10 21:14:31.750313 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.750262    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa9f257-4081-4e30-9005-74f29e2f87d3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cfa9f257-4081-4e30-9005-74f29e2f87d3" (UID: "cfa9f257-4081-4e30-9005-74f29e2f87d3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:31.752310 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.752281    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa9f257-4081-4e30-9005-74f29e2f87d3-kube-api-access-tw65v" (OuterVolumeSpecName: "kube-api-access-tw65v") pod "cfa9f257-4081-4e30-9005-74f29e2f87d3" (UID: "cfa9f257-4081-4e30-9005-74f29e2f87d3"). InnerVolumeSpecName "kube-api-access-tw65v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:31.752603 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.752580    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cfa9f257-4081-4e30-9005-74f29e2f87d3" (UID: "cfa9f257-4081-4e30-9005-74f29e2f87d3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:31.752690 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.752634    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cfa9f257-4081-4e30-9005-74f29e2f87d3" (UID: "cfa9f257-4081-4e30-9005-74f29e2f87d3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:14:31.760946 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.760926    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"]
Sep 10 21:14:31.761215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.761187    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy" containerID="cri-o://6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e" gracePeriod=30
Sep 10 21:14:31.851280 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.851229    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa9f257-4081-4e30-9005-74f29e2f87d3-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:31.851280 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.851255    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:31.851280 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.851264    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw65v\" (UniqueName: \"kubernetes.io/projected/cfa9f257-4081-4e30-9005-74f29e2f87d3-kube-api-access-tw65v\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:31.851280 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:31.851272    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cfa9f257-4081-4e30-9005-74f29e2f87d3-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:32.471795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.471762    2580 generic.go:356] "Generic (PLEG): container finished" podID="cfa9f257-4081-4e30-9005-74f29e2f87d3" containerID="a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0" exitCode=0
Sep 10 21:14:32.472227 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.471804    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r" event={"ID":"cfa9f257-4081-4e30-9005-74f29e2f87d3","Type":"ContainerDied","Data":"a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0"}
Sep 10 21:14:32.472227 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.471810    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"
Sep 10 21:14:32.472227 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.471846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r" event={"ID":"cfa9f257-4081-4e30-9005-74f29e2f87d3","Type":"ContainerDied","Data":"211f1a096a030f53ca3ef818d5c6ee19cdd1e7d2677887faf71e7dcc45b8acff"}
Sep 10 21:14:32.472227 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.471867    2580 scope.go:117] "RemoveContainer" containerID="a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0"
Sep 10 21:14:32.482338 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.482319    2580 scope.go:117] "RemoveContainer" containerID="a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0"
Sep 10 21:14:32.482613 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:14:32.482592    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0\": container with ID starting with a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0 not found: ID does not exist" containerID="a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0"
Sep 10 21:14:32.482676 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.482621    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0"} err="failed to get container status \"a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0\": rpc error: code = NotFound desc = could not find container \"a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0\": container with ID starting with a464a02f1a8d752d7b1ebd07cce1c62cfa8f6b9b69733174941f120e147160b0 not found: ID does not exist"
Sep 10 21:14:32.499999 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.499946    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"]
Sep 10 21:14:32.503134 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:32.503104    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-3002a381-6b47d9cb7c-66k7r"]
Sep 10 21:14:33.519875 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:33.519842    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa9f257-4081-4e30-9005-74f29e2f87d3" path="/var/lib/kubelet/pods/cfa9f257-4081-4e30-9005-74f29e2f87d3/volumes"
Sep 10 21:14:35.546222 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:35.546187    2580 patch_prober.go:28] interesting pod/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:14:35.546691 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:35.546261    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:14:37.061473 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.061402    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:14:37.200491 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200462    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-podinfo\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.200694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200499    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-envoy\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.200694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200530    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-token\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.200694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200657    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399c36f5-82d2-4c10-ab6a-604cb3084d88-istiod-ca-cert\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.200864 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200702    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-credential-socket\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.200864 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200731    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j6kfl\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-kube-api-access-j6kfl\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.200864 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200764    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-socket\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.200864 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200792    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-data\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.200864 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200808    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-certs\") pod \"399c36f5-82d2-4c10-ab6a-604cb3084d88\" (UID: \"399c36f5-82d2-4c10-ab6a-604cb3084d88\") "
Sep 10 21:14:37.201138 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.200898    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:37.201138 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.201014    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/399c36f5-82d2-4c10-ab6a-604cb3084d88-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:14:37.201138 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.201051    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:37.201464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.201177    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-data" (OuterVolumeSpecName: "istio-data") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:37.201464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.201229    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/399c36f5-82d2-4c10-ab6a-604cb3084d88-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.201464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.201229    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:37.201464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.201246    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.201464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.201260    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.202998 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.202964    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:14:37.203148 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.203016    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:14:37.203148 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.203035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-token" (OuterVolumeSpecName: "istio-token") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:37.203273 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.203216    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-kube-api-access-j6kfl" (OuterVolumeSpecName: "kube-api-access-j6kfl") pod "399c36f5-82d2-4c10-ab6a-604cb3084d88" (UID: "399c36f5-82d2-4c10-ab6a-604cb3084d88"). InnerVolumeSpecName "kube-api-access-j6kfl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:14:37.302579 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.302551    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.302579 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.302577    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.302579 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.302586    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.302829 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.302594    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6kfl\" (UniqueName: \"kubernetes.io/projected/399c36f5-82d2-4c10-ab6a-604cb3084d88-kube-api-access-j6kfl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.302829 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.302605    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.302829 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.302614    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/399c36f5-82d2-4c10-ab6a-604cb3084d88-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:14:37.505397 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.505360    2580 generic.go:356] "Generic (PLEG): container finished" podID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerID="6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e" exitCode=0
Sep 10 21:14:37.505596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.505417    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"
Sep 10 21:14:37.505596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.505427    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" event={"ID":"399c36f5-82d2-4c10-ab6a-604cb3084d88","Type":"ContainerDied","Data":"6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e"}
Sep 10 21:14:37.505596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.505470    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d" event={"ID":"399c36f5-82d2-4c10-ab6a-604cb3084d88","Type":"ContainerDied","Data":"11c4b2974edf870742ed01a5e8bc3c3dc34747c21382326fe245dd5be0cfb839"}
Sep 10 21:14:37.505596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.505492    2580 scope.go:117] "RemoveContainer" containerID="6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e"
Sep 10 21:14:37.515438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.515422    2580 scope.go:117] "RemoveContainer" containerID="02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887"
Sep 10 21:14:37.524600 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.524584    2580 scope.go:117] "RemoveContainer" containerID="6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e"
Sep 10 21:14:37.524861 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:14:37.524844    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e\": container with ID starting with 6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e not found: ID does not exist" containerID="6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e"
Sep 10 21:14:37.524905 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.524871    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e"} err="failed to get container status \"6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e\": rpc error: code = NotFound desc = could not find container \"6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e\": container with ID starting with 6b07df3ff67e583132c76e1dfa93b72cf0e62dcdd45327b3c893d924c6e4de6e not found: ID does not exist"
Sep 10 21:14:37.524905 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.524888    2580 scope.go:117] "RemoveContainer" containerID="02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887"
Sep 10 21:14:37.525168 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:14:37.525151    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887\": container with ID starting with 02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887 not found: ID does not exist" containerID="02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887"
Sep 10 21:14:37.525210 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.525175    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887"} err="failed to get container status \"02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887\": rpc error: code = NotFound desc = could not find container \"02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887\": container with ID starting with 02c0213a54d49de76ecfec7e9fe8f803bd2d7a375bce38b58168540d7b81b887 not found: ID does not exist"
Sep 10 21:14:37.536697 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.536677    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"]
Sep 10 21:14:37.546112 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:37.546093    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-3002a381-openshift-default-79d49f74fb-grj9d"]
Sep 10 21:14:39.520068 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:14:39.520033    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" path="/var/lib/kubelet/pods/399c36f5-82d2-4c10-ab6a-604cb3084d88/volumes"
Sep 10 21:15:16.593324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:16.593288    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"]
Sep 10 21:15:16.647477 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:15:16.647448    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-serving-cert: secret "maas-api-e2e-explicit-2bbc78-serving-cert" not found
Sep 10 21:15:16.647772 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:15:16.647738    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls podName:80a5b171-e02a-4863-8740-9acd5c6b2d44 nodeName:}" failed. No retries permitted until 2026-09-10 21:15:17.147710829 +0000 UTC m=+1004.149957170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls") pod "maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4" (UID: "80a5b171-e02a-4863-8740-9acd5c6b2d44") : secret "maas-api-e2e-explicit-2bbc78-serving-cert" not found
Sep 10 21:15:16.663753 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:16.663719    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"]
Sep 10 21:15:16.664012 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:16.663963    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb" podUID="9f664264-24c5-4e6c-b9b5-107cabf5f812" containerName="payload-pre-processing" containerID="cri-o://ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87" gracePeriod=30
Sep 10 21:15:16.724681 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:16.724645    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4" podUID="80a5b171-e02a-4863-8740-9acd5c6b2d44" containerName="maas-api" containerID="cri-o://1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a" gracePeriod=30
Sep 10 21:15:16.989545 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:16.989517    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:15:17.002183 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.002163    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:15:17.151480 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.151399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls\") pod \"80a5b171-e02a-4863-8740-9acd5c6b2d44\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") "
Sep 10 21:15:17.151642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.151487    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6fhv\" (UniqueName: \"kubernetes.io/projected/9f664264-24c5-4e6c-b9b5-107cabf5f812-kube-api-access-v6fhv\") pod \"9f664264-24c5-4e6c-b9b5-107cabf5f812\" (UID: \"9f664264-24c5-4e6c-b9b5-107cabf5f812\") "
Sep 10 21:15:17.151642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.151516    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-metrics-certs\") pod \"80a5b171-e02a-4863-8740-9acd5c6b2d44\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") "
Sep 10 21:15:17.151642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.151531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dp85j\" (UniqueName: \"kubernetes.io/projected/80a5b171-e02a-4863-8740-9acd5c6b2d44-kube-api-access-dp85j\") pod \"80a5b171-e02a-4863-8740-9acd5c6b2d44\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") "
Sep 10 21:15:17.151642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.151554    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f664264-24c5-4e6c-b9b5-107cabf5f812-plugins-config-volume\") pod \"9f664264-24c5-4e6c-b9b5-107cabf5f812\" (UID: \"9f664264-24c5-4e6c-b9b5-107cabf5f812\") "
Sep 10 21:15:17.151642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.151591    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a5b171-e02a-4863-8740-9acd5c6b2d44-openshift-service-ca\") pod \"80a5b171-e02a-4863-8740-9acd5c6b2d44\" (UID: \"80a5b171-e02a-4863-8740-9acd5c6b2d44\") "
Sep 10 21:15:17.151971 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.151940    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f664264-24c5-4e6c-b9b5-107cabf5f812-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f664264-24c5-4e6c-b9b5-107cabf5f812" (UID: "9f664264-24c5-4e6c-b9b5-107cabf5f812"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:15:17.152036 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.151958    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80a5b171-e02a-4863-8740-9acd5c6b2d44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "80a5b171-e02a-4863-8740-9acd5c6b2d44" (UID: "80a5b171-e02a-4863-8740-9acd5c6b2d44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:15:17.153724 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.153700    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "80a5b171-e02a-4863-8740-9acd5c6b2d44" (UID: "80a5b171-e02a-4863-8740-9acd5c6b2d44"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:17.153813 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.153731    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "80a5b171-e02a-4863-8740-9acd5c6b2d44" (UID: "80a5b171-e02a-4863-8740-9acd5c6b2d44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:15:17.154245 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.154221    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f664264-24c5-4e6c-b9b5-107cabf5f812-kube-api-access-v6fhv" (OuterVolumeSpecName: "kube-api-access-v6fhv") pod "9f664264-24c5-4e6c-b9b5-107cabf5f812" (UID: "9f664264-24c5-4e6c-b9b5-107cabf5f812"). InnerVolumeSpecName "kube-api-access-v6fhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:17.154331 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.154256    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80a5b171-e02a-4863-8740-9acd5c6b2d44-kube-api-access-dp85j" (OuterVolumeSpecName: "kube-api-access-dp85j") pod "80a5b171-e02a-4863-8740-9acd5c6b2d44" (UID: "80a5b171-e02a-4863-8740-9acd5c6b2d44"). InnerVolumeSpecName "kube-api-access-dp85j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:15:17.252803 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.252764    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:15:17.252803 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.252797    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6fhv\" (UniqueName: \"kubernetes.io/projected/9f664264-24c5-4e6c-b9b5-107cabf5f812-kube-api-access-v6fhv\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:15:17.252803 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.252807    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80a5b171-e02a-4863-8740-9acd5c6b2d44-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:15:17.253052 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.252815    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dp85j\" (UniqueName: \"kubernetes.io/projected/80a5b171-e02a-4863-8740-9acd5c6b2d44-kube-api-access-dp85j\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:15:17.253052 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.252824    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f664264-24c5-4e6c-b9b5-107cabf5f812-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:15:17.253052 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.252832    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80a5b171-e02a-4863-8740-9acd5c6b2d44-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:15:17.731459 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.731429    2580 generic.go:356] "Generic (PLEG): container finished" podID="80a5b171-e02a-4863-8740-9acd5c6b2d44" containerID="1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a" exitCode=0
Sep 10 21:15:17.731910 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.731479    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"
Sep 10 21:15:17.731910 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.731510    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4" event={"ID":"80a5b171-e02a-4863-8740-9acd5c6b2d44","Type":"ContainerDied","Data":"1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a"}
Sep 10 21:15:17.731910 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.731563    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4" event={"ID":"80a5b171-e02a-4863-8740-9acd5c6b2d44","Type":"ContainerDied","Data":"27c6b61b9a25236f25365defa4088444f69bdbe9b38124245131a662cb870a3c"}
Sep 10 21:15:17.731910 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.731584    2580 scope.go:117] "RemoveContainer" containerID="1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a"
Sep 10 21:15:17.733093 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.733018    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f664264-24c5-4e6c-b9b5-107cabf5f812" containerID="ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87" exitCode=0
Sep 10 21:15:17.733093 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.733061    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"
Sep 10 21:15:17.733263 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.733106    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb" event={"ID":"9f664264-24c5-4e6c-b9b5-107cabf5f812","Type":"ContainerDied","Data":"ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87"}
Sep 10 21:15:17.733381 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.733295    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb" event={"ID":"9f664264-24c5-4e6c-b9b5-107cabf5f812","Type":"ContainerDied","Data":"8f81ead44fb052a371a0819aa2b5d70f47df67b96fa6271cc344e857c6457994"}
Sep 10 21:15:17.741213 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.741196    2580 scope.go:117] "RemoveContainer" containerID="1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a"
Sep 10 21:15:17.741470 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:15:17.741453    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a\": container with ID starting with 1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a not found: ID does not exist" containerID="1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a"
Sep 10 21:15:17.741551 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.741483    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a"} err="failed to get container status \"1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a\": rpc error: code = NotFound desc = could not find container \"1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a\": container with ID starting with 1d4fa3c5912e2c4a1fc7e00ddc02a5b7a97f77976068f1ae869ae4d93d74fa1a not found: ID does not exist"
Sep 10 21:15:17.741551 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.741504    2580 scope.go:117] "RemoveContainer" containerID="ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87"
Sep 10 21:15:17.749742 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.749724    2580 scope.go:117] "RemoveContainer" containerID="ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87"
Sep 10 21:15:17.750013 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:15:17.749991    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87\": container with ID starting with ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87 not found: ID does not exist" containerID="ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87"
Sep 10 21:15:17.750080 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.750017    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87"} err="failed to get container status \"ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87\": rpc error: code = NotFound desc = could not find container \"ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87\": container with ID starting with ecf8f68507cbe9bc93c5ec62ed52acaf5cb4f08826fad342693c1fd22c4d7d87 not found: ID does not exist"
Sep 10 21:15:17.750355 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.750337    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"]
Sep 10 21:15:17.753920 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.753899    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-2bbc78-85fd6c6566-4r5mb"]
Sep 10 21:15:17.766589 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.766567    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"]
Sep 10 21:15:17.769796 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:17.769779    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-2bbc78-68b8f56cf6-w5ct4"]
Sep 10 21:15:19.519518 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:19.519484    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80a5b171-e02a-4863-8740-9acd5c6b2d44" path="/var/lib/kubelet/pods/80a5b171-e02a-4863-8740-9acd5c6b2d44/volumes"
Sep 10 21:15:19.519886 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:19.519862    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f664264-24c5-4e6c-b9b5-107cabf5f812" path="/var/lib/kubelet/pods/9f664264-24c5-4e6c-b9b5-107cabf5f812/volumes"
Sep 10 21:15:24.579379 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579346    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"]
Sep 10 21:15:24.579777 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579764    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy"
Sep 10 21:15:24.579827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579779    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy"
Sep 10 21:15:24.579827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579789    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy"
Sep 10 21:15:24.579827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579794    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy"
Sep 10 21:15:24.579827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579800    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f664264-24c5-4e6c-b9b5-107cabf5f812" containerName="payload-pre-processing"
Sep 10 21:15:24.579827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579805    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f664264-24c5-4e6c-b9b5-107cabf5f812" containerName="payload-pre-processing"
Sep 10 21:15:24.579827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579811    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80a5b171-e02a-4863-8740-9acd5c6b2d44" containerName="maas-api"
Sep 10 21:15:24.579827 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579817    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="80a5b171-e02a-4863-8740-9acd5c6b2d44" containerName="maas-api"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579838    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579844    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579850    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa9f257-4081-4e30-9005-74f29e2f87d3" containerName="maas-api"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579855    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa9f257-4081-4e30-9005-74f29e2f87d3" containerName="maas-api"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579918    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa9f257-4081-4e30-9005-74f29e2f87d3" containerName="maas-api"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579928    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579934    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f664264-24c5-4e6c-b9b5-107cabf5f812" containerName="payload-pre-processing"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579939    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579945    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="80a5b171-e02a-4863-8740-9acd5c6b2d44" containerName="maas-api"
Sep 10 21:15:24.580065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.579950    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="399c36f5-82d2-4c10-ab6a-604cb3084d88" containerName="istio-proxy"
Sep 10 21:15:24.584617 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.584598    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.586854 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.586830    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:15:24.586982 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.586896    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-fbb03d-openshift-default-dockercfg-h6glg\""
Sep 10 21:15:24.595552 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.595529    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"]
Sep 10 21:15:24.722509 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-token\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.722509 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-credential-socket\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.722733 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722551    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-envoy\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.722733 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722629    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787584-fa2b-447d-88be-8dd05bb0e45a-istiod-ca-cert\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.722733 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722661    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wjv\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-kube-api-access-79wjv\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.722733 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722683    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-podinfo\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.722887 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722738    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-data\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.722887 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722776    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-socket\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.722887 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.722811    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-certs\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.823844 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.823808    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787584-fa2b-447d-88be-8dd05bb0e45a-istiod-ca-cert\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.823844 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.823845    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wjv\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-kube-api-access-79wjv\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824098 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.823867    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-podinfo\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824098 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.823888    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-data\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824098 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.823929    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-socket\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824098 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.823973    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-certs\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824098 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.824012    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-token\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824098 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.824040    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-credential-socket\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824098 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.824096    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-envoy\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.824326    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-data\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.824384    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-socket\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.824407    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-certs\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.824507    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-credential-socket\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.824565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.824565    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787584-fa2b-447d-88be-8dd05bb0e45a-istiod-ca-cert\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.826555 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.826532    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-podinfo\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.826653 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.826536    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-envoy\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.831488 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.831439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-token\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.831771 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.831752    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wjv\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-kube-api-access-79wjv\") pod \"e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") " pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:24.895995 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:24.895971    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:25.183415 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:25.183374    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"]
Sep 10 21:15:25.184374 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:15:25.184343    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e787584_fa2b_447d_88be_8dd05bb0e45a.slice/crio-2246de7a008a1e9dbd8a2796a47318af95809739350f238bd7e0f46f5eb9dd28 WatchSource:0}: Error finding container 2246de7a008a1e9dbd8a2796a47318af95809739350f238bd7e0f46f5eb9dd28: Status 404 returned error can't find the container with id 2246de7a008a1e9dbd8a2796a47318af95809739350f238bd7e0f46f5eb9dd28
Sep 10 21:15:25.187096 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:25.187065    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:15:25.187193 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:25.187148    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:15:25.187193 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:25.187178    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:15:25.781363 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:25.781330    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z" event={"ID":"4e787584-fa2b-447d-88be-8dd05bb0e45a","Type":"ContainerStarted","Data":"e07449a262571e5cdb0b23b991a3d251f574670e31a30cd558ca53e3feccd4c5"}
Sep 10 21:15:25.781741 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:25.781384    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z" event={"ID":"4e787584-fa2b-447d-88be-8dd05bb0e45a","Type":"ContainerStarted","Data":"2246de7a008a1e9dbd8a2796a47318af95809739350f238bd7e0f46f5eb9dd28"}
Sep 10 21:15:25.802504 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:25.802454    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z" podStartSLOduration=1.802437085 podStartE2EDuration="1.802437085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:25.801410687 +0000 UTC m=+1012.803657035" watchObservedRunningTime="2026-09-10 21:15:25.802437085 +0000 UTC m=+1012.804683432"
Sep 10 21:15:25.897077 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:25.897040    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:26.902857 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:26.902828    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:27.792822 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:27.792780    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:27.793897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:27.793881    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:15:50.773465 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.773423    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"]
Sep 10 21:15:50.774061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.773830    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy"
Sep 10 21:15:50.774061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.773843    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy"
Sep 10 21:15:50.774061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.773911    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="34aaaf9a-378f-4ce0-9936-d53fd9e4b099" containerName="istio-proxy"
Sep 10 21:15:50.778616 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.778594    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.782209 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.781895    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-fbb03d\"/\"noten-model-fbb03d-kserve-self-signed-certs\""
Sep 10 21:15:50.782361 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.781993    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-fbb03d\"/\"default-dockercfg-df9n9\""
Sep 10 21:15:50.782445 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.781899    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-fbb03d\"/\"openshift-service-ca.crt\""
Sep 10 21:15:50.782506 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.782191    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-fbb03d\"/\"kube-root-ca.crt\""
Sep 10 21:15:50.785356 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.785332    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"]
Sep 10 21:15:50.847951 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.847918    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-home\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.848090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.847962    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-dshm\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.848090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.847983    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-model-cache\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.848090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.848000    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7ttl\" (UniqueName: \"kubernetes.io/projected/01d78851-5906-4fb2-8397-9378e5ff876d-kube-api-access-r7ttl\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.848230 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.848082    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d78851-5906-4fb2-8397-9378e5ff876d-tls-certs\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.848230 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.848110    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-tmp-dir\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949152 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949104    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-home\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949167    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-dshm\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-model-cache\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949206    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7ttl\" (UniqueName: \"kubernetes.io/projected/01d78851-5906-4fb2-8397-9378e5ff876d-kube-api-access-r7ttl\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949240    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d78851-5906-4fb2-8397-9378e5ff876d-tls-certs\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949255    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-tmp-dir\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949509    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-home\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949596 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949579    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-model-cache\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.949694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.949645    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-tmp-dir\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.951629 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.951606    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-dshm\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.951882 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.951865    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d78851-5906-4fb2-8397-9378e5ff876d-tls-certs\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:50.958153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:50.958112    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7ttl\" (UniqueName: \"kubernetes.io/projected/01d78851-5906-4fb2-8397-9378e5ff876d-kube-api-access-r7ttl\") pod \"noten-model-fbb03d-kserve-c988d48c-h44pk\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") " pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:51.097564 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:51.097481    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:15:51.376675 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:51.376652    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"]
Sep 10 21:15:51.380304 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:15:51.380249    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01d78851_5906_4fb2_8397_9378e5ff876d.slice/crio-3a664c1ca2923ae69d2d55a4fcc606c7bb9c2dd2e14ef774cb5e6cee68ea50f0 WatchSource:0}: Error finding container 3a664c1ca2923ae69d2d55a4fcc606c7bb9c2dd2e14ef774cb5e6cee68ea50f0: Status 404 returned error can't find the container with id 3a664c1ca2923ae69d2d55a4fcc606c7bb9c2dd2e14ef774cb5e6cee68ea50f0
Sep 10 21:15:51.935192 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:51.935147    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk" event={"ID":"01d78851-5906-4fb2-8397-9378e5ff876d","Type":"ContainerStarted","Data":"1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404"}
Sep 10 21:15:51.935192 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:51.935196    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk" event={"ID":"01d78851-5906-4fb2-8397-9378e5ff876d","Type":"ContainerStarted","Data":"3a664c1ca2923ae69d2d55a4fcc606c7bb9c2dd2e14ef774cb5e6cee68ea50f0"}
Sep 10 21:15:51.957058 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:51.957011    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk" podStartSLOduration=1.9569955810000002 podStartE2EDuration="1.956995581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:15:51.95422476 +0000 UTC m=+1038.956471110" watchObservedRunningTime="2026-09-10 21:15:51.956995581 +0000 UTC m=+1038.959241927"
Sep 10 21:15:56.714131 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:56.714082    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"]
Sep 10 21:15:56.866648 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:56.866612    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"]
Sep 10 21:15:56.866875 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:56.866832    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z" podUID="4e787584-fa2b-447d-88be-8dd05bb0e45a" containerName="istio-proxy" containerID="cri-o://e07449a262571e5cdb0b23b991a3d251f574670e31a30cd558ca53e3feccd4c5" gracePeriod=30
Sep 10 21:15:57.795964 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:57.795924    2580 patch_prober.go:28] interesting pod/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:15:57.796361 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:15:57.795995    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z" podUID="4e787584-fa2b-447d-88be-8dd05bb0e45a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:16:01.098495 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:01.098462    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:16:01.991616 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:01.991588    2580 generic.go:356] "Generic (PLEG): container finished" podID="4e787584-fa2b-447d-88be-8dd05bb0e45a" containerID="e07449a262571e5cdb0b23b991a3d251f574670e31a30cd558ca53e3feccd4c5" exitCode=0
Sep 10 21:16:01.991747 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:01.991645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z" event={"ID":"4e787584-fa2b-447d-88be-8dd05bb0e45a","Type":"ContainerDied","Data":"e07449a262571e5cdb0b23b991a3d251f574670e31a30cd558ca53e3feccd4c5"}
Sep 10 21:16:02.123104 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.123081    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:16:02.252982 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.252903    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-envoy\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.252982 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.252960    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787584-fa2b-447d-88be-8dd05bb0e45a-istiod-ca-cert\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.253255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.252991    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-credential-socket\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.253255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253008    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-podinfo\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.253255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253030    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-socket\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.253255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253070    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-data\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.253255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253145    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-certs\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.253255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253177    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-token\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.253255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253214    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79wjv\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-kube-api-access-79wjv\") pod \"4e787584-fa2b-447d-88be-8dd05bb0e45a\" (UID: \"4e787584-fa2b-447d-88be-8dd05bb0e45a\") "
Sep 10 21:16:02.253625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253324    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:02.253625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253395    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:02.253625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253416    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:02.253625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253586    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:02.253625 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253612    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e787584-fa2b-447d-88be-8dd05bb0e45a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:02.254064 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253656    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.254064 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253671    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.254064 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.253680    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.255277 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.255245    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:02.255424 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.255404    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-kube-api-access-79wjv" (OuterVolumeSpecName: "kube-api-access-79wjv") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "kube-api-access-79wjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:02.255589 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.255564    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:16:02.255674 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.255646    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e787584-fa2b-447d-88be-8dd05bb0e45a" (UID: "4e787584-fa2b-447d-88be-8dd05bb0e45a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:02.354103 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.354073    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e787584-fa2b-447d-88be-8dd05bb0e45a-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.354103 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.354096    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.354103 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.354105    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.354314 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.354114    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.354314 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.354143    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wjv\" (UniqueName: \"kubernetes.io/projected/4e787584-fa2b-447d-88be-8dd05bb0e45a-kube-api-access-79wjv\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.354314 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.354156    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e787584-fa2b-447d-88be-8dd05bb0e45a-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:02.998364 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.998332    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z" event={"ID":"4e787584-fa2b-447d-88be-8dd05bb0e45a","Type":"ContainerDied","Data":"2246de7a008a1e9dbd8a2796a47318af95809739350f238bd7e0f46f5eb9dd28"}
Sep 10 21:16:02.998364 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.998362    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"
Sep 10 21:16:02.998620 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:02.998395    2580 scope.go:117] "RemoveContainer" containerID="e07449a262571e5cdb0b23b991a3d251f574670e31a30cd558ca53e3feccd4c5"
Sep 10 21:16:03.028389 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:03.028358    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"]
Sep 10 21:16:03.030455 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:03.030422    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-fbb03d-openshift-default-6f7bbfb9fd-bs49z"]
Sep 10 21:16:03.521358 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:03.521323    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e787584-fa2b-447d-88be-8dd05bb0e45a" path="/var/lib/kubelet/pods/4e787584-fa2b-447d-88be-8dd05bb0e45a/volumes"
Sep 10 21:16:10.787764 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.787728    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"]
Sep 10 21:16:10.788285 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.788265    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e787584-fa2b-447d-88be-8dd05bb0e45a" containerName="istio-proxy"
Sep 10 21:16:10.788362 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.788287    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e787584-fa2b-447d-88be-8dd05bb0e45a" containerName="istio-proxy"
Sep 10 21:16:10.788419 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.788409    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e787584-fa2b-447d-88be-8dd05bb0e45a" containerName="istio-proxy"
Sep 10 21:16:10.793297 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.793274    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.795587 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.795566    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:16:10.795587 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.795577    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-170a718d-openshift-default-dockercfg-m4shl\""
Sep 10 21:16:10.804047 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.804026    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"]
Sep 10 21:16:10.927307 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927275    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-envoy\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.927463 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927312    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-certs\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.927463 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927341    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istiod-ca-cert\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.927463 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927418    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-data\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.927572 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927468    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhxm\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-kube-api-access-2zhxm\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.927572 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927493    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-credential-socket\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.927572 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927512    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-socket\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.927572 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-token\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:10.927699 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:10.927602    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-podinfo\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.028727 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.028691    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhxm\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-kube-api-access-2zhxm\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.028727 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.028729    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-credential-socket\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.028970 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.028745    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-socket\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.028970 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.028936    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-token\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029084 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029007    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-podinfo\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029084 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029031    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-envoy\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029084 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029056    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-certs\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029292 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029103    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istiod-ca-cert\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029292 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029171    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-socket\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029292 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029190    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-data\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029292 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029106    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-credential-socket\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029459 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029439    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-certs\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029531 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029517    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-data\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.029769 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.029747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istiod-ca-cert\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.031400 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.031379    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-envoy\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.031602 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.031585    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-podinfo\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.037865 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.037803    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-token\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.038144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.038093    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhxm\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-kube-api-access-2zhxm\") pod \"e2e-mt-170a718d-openshift-default-84789669b4-fqfj4\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") " pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.105456 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.105424    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:11.383806 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.383783    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"]
Sep 10 21:16:11.385798 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:16:11.385754    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bca6d2e_05dd_48e4_9533_0587c5949e0a.slice/crio-b8394121f90083c6a67220770d94c151e6da046209693413b518bc5bcb1f8f0d WatchSource:0}: Error finding container b8394121f90083c6a67220770d94c151e6da046209693413b518bc5bcb1f8f0d: Status 404 returned error can't find the container with id b8394121f90083c6a67220770d94c151e6da046209693413b518bc5bcb1f8f0d
Sep 10 21:16:11.388140 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.388086    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:16:11.388209 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.388185    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:16:11.388248 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.388224    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:16:11.734217 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.734099    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk" podUID="01d78851-5906-4fb2-8397-9378e5ff876d" containerName="main" containerID="cri-o://1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404" gracePeriod=45
Sep 10 21:16:11.971926 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:11.971900    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:16:12.054648 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.054607    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" event={"ID":"7bca6d2e-05dd-48e4-9533-0587c5949e0a","Type":"ContainerStarted","Data":"4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865"}
Sep 10 21:16:12.054816 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.054673    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" event={"ID":"7bca6d2e-05dd-48e4-9533-0587c5949e0a","Type":"ContainerStarted","Data":"b8394121f90083c6a67220770d94c151e6da046209693413b518bc5bcb1f8f0d"}
Sep 10 21:16:12.056016 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.055987    2580 generic.go:356] "Generic (PLEG): container finished" podID="01d78851-5906-4fb2-8397-9378e5ff876d" containerID="1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404" exitCode=0
Sep 10 21:16:12.056172 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.056025    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"
Sep 10 21:16:12.056172 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.056036    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk" event={"ID":"01d78851-5906-4fb2-8397-9378e5ff876d","Type":"ContainerDied","Data":"1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404"}
Sep 10 21:16:12.056172 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.056071    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk" event={"ID":"01d78851-5906-4fb2-8397-9378e5ff876d","Type":"ContainerDied","Data":"3a664c1ca2923ae69d2d55a4fcc606c7bb9c2dd2e14ef774cb5e6cee68ea50f0"}
Sep 10 21:16:12.056172 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.056095    2580 scope.go:117] "RemoveContainer" containerID="1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404"
Sep 10 21:16:12.064928 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.064912    2580 scope.go:117] "RemoveContainer" containerID="1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404"
Sep 10 21:16:12.065221 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:16:12.065201    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404\": container with ID starting with 1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404 not found: ID does not exist" containerID="1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404"
Sep 10 21:16:12.065279 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.065230    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404"} err="failed to get container status \"1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404\": rpc error: code = NotFound desc = could not find container \"1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404\": container with ID starting with 1a211b8987c1f14d5bf4b5128d10d9f2aa933efb4f255e3329504f772fbbc404 not found: ID does not exist"
Sep 10 21:16:12.076912 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.076840    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podStartSLOduration=2.076823843 podStartE2EDuration="2.076823843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:12.075473284 +0000 UTC m=+1059.077719631" watchObservedRunningTime="2026-09-10 21:16:12.076823843 +0000 UTC m=+1059.079070192"
Sep 10 21:16:12.105590 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.105572    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:12.110467 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.110447    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:12.140893 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.140869    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7ttl\" (UniqueName: \"kubernetes.io/projected/01d78851-5906-4fb2-8397-9378e5ff876d-kube-api-access-r7ttl\") pod \"01d78851-5906-4fb2-8397-9378e5ff876d\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") "
Sep 10 21:16:12.141065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.140904    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-model-cache\") pod \"01d78851-5906-4fb2-8397-9378e5ff876d\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") "
Sep 10 21:16:12.141065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.140929    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d78851-5906-4fb2-8397-9378e5ff876d-tls-certs\") pod \"01d78851-5906-4fb2-8397-9378e5ff876d\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") "
Sep 10 21:16:12.141065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.140958    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-dshm\") pod \"01d78851-5906-4fb2-8397-9378e5ff876d\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") "
Sep 10 21:16:12.141065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.141036    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-tmp-dir\") pod \"01d78851-5906-4fb2-8397-9378e5ff876d\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") "
Sep 10 21:16:12.141332 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.141082    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-home\") pod \"01d78851-5906-4fb2-8397-9378e5ff876d\" (UID: \"01d78851-5906-4fb2-8397-9378e5ff876d\") "
Sep 10 21:16:12.141332 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.141209    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-model-cache" (OuterVolumeSpecName: "model-cache") pod "01d78851-5906-4fb2-8397-9378e5ff876d" (UID: "01d78851-5906-4fb2-8397-9378e5ff876d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.141332 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.141310    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01d78851-5906-4fb2-8397-9378e5ff876d" (UID: "01d78851-5906-4fb2-8397-9378e5ff876d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.141493 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.141335    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-home" (OuterVolumeSpecName: "home") pod "01d78851-5906-4fb2-8397-9378e5ff876d" (UID: "01d78851-5906-4fb2-8397-9378e5ff876d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.141550 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.141503    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.141550 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.141523    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.141550 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.141537    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.143365 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.143341    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-dshm" (OuterVolumeSpecName: "dshm") pod "01d78851-5906-4fb2-8397-9378e5ff876d" (UID: "01d78851-5906-4fb2-8397-9378e5ff876d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:16:12.143468 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.143394    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d78851-5906-4fb2-8397-9378e5ff876d-kube-api-access-r7ttl" (OuterVolumeSpecName: "kube-api-access-r7ttl") pod "01d78851-5906-4fb2-8397-9378e5ff876d" (UID: "01d78851-5906-4fb2-8397-9378e5ff876d"). InnerVolumeSpecName "kube-api-access-r7ttl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:12.143519 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.143460    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d78851-5906-4fb2-8397-9378e5ff876d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01d78851-5906-4fb2-8397-9378e5ff876d" (UID: "01d78851-5906-4fb2-8397-9378e5ff876d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:12.242379 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.242348    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01d78851-5906-4fb2-8397-9378e5ff876d-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.242379 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.242378    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01d78851-5906-4fb2-8397-9378e5ff876d-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.242577 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.242390    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7ttl\" (UniqueName: \"kubernetes.io/projected/01d78851-5906-4fb2-8397-9378e5ff876d-kube-api-access-r7ttl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:12.379840 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.379782    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"]
Sep 10 21:16:12.382514 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:12.382488    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-fbb03d/noten-model-fbb03d-kserve-c988d48c-h44pk"]
Sep 10 21:16:13.064216 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:13.064183    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:13.065020 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:13.065002    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:13.520357 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:13.520326    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d78851-5906-4fb2-8397-9378e5ff876d" path="/var/lib/kubelet/pods/01d78851-5906-4fb2-8397-9378e5ff876d/volumes"
Sep 10 21:16:15.132638 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.132606    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"]
Sep 10 21:16:15.133109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.132976    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d78851-5906-4fb2-8397-9378e5ff876d" containerName="main"
Sep 10 21:16:15.133109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.132987    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d78851-5906-4fb2-8397-9378e5ff876d" containerName="main"
Sep 10 21:16:15.133109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.133051    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d78851-5906-4fb2-8397-9378e5ff876d" containerName="main"
Sep 10 21:16:15.137360 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.137338    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.139946 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.139924    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-170a718d-serving-cert\""
Sep 10 21:16:15.150283 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.150260    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"]
Sep 10 21:16:15.268679 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.268649    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5827f1-1262-4afe-be05-91c93daab648-openshift-service-ca\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.268679 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.268690    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2xvx\" (UniqueName: \"kubernetes.io/projected/6e5827f1-1262-4afe-be05-91c93daab648-kube-api-access-f2xvx\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.268896 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.268747    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-maas-api-tls\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.268896 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.268767    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-metrics-certs\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.369370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.369334    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5827f1-1262-4afe-be05-91c93daab648-openshift-service-ca\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.369370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.369375    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2xvx\" (UniqueName: \"kubernetes.io/projected/6e5827f1-1262-4afe-be05-91c93daab648-kube-api-access-f2xvx\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.369583 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.369420    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-maas-api-tls\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.369583 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.369441    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-metrics-certs\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.370156 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.370033    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5827f1-1262-4afe-be05-91c93daab648-openshift-service-ca\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.372065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.372040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-maas-api-tls\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.372198 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.372090    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-metrics-certs\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.379683 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.379654    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2xvx\" (UniqueName: \"kubernetes.io/projected/6e5827f1-1262-4afe-be05-91c93daab648-kube-api-access-f2xvx\") pod \"maas-api-e2e-mt-170a718d-8656bc6596-vz4fm\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.449332 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.449257    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:15.744806 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.744782    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"]
Sep 10 21:16:15.746796 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:16:15.746765    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5827f1_1262_4afe_be05_91c93daab648.slice/crio-3f2079dd76a8e39456173976197df6f9d961f084d25ef608bc73f1c94564bb96 WatchSource:0}: Error finding container 3f2079dd76a8e39456173976197df6f9d961f084d25ef608bc73f1c94564bb96: Status 404 returned error can't find the container with id 3f2079dd76a8e39456173976197df6f9d961f084d25ef608bc73f1c94564bb96
Sep 10 21:16:15.879829 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.879794    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"]
Sep 10 21:16:15.884989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.884955    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:15.887618 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.887596    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-170a718d\""
Sep 10 21:16:15.891876 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.891848    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"]
Sep 10 21:16:15.975533 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.975500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwkl\" (UniqueName: \"kubernetes.io/projected/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-kube-api-access-7xwkl\") pod \"payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz\" (UID: \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\") " pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:15.975698 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:15.975547    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-plugins-config-volume\") pod \"payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz\" (UID: \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\") " pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:16.076372 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:16.076272    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwkl\" (UniqueName: \"kubernetes.io/projected/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-kube-api-access-7xwkl\") pod \"payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz\" (UID: \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\") " pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:16.076372 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:16.076341    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-plugins-config-volume\") pod \"payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz\" (UID: \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\") " pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:16.077319 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:16.077283    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-plugins-config-volume\") pod \"payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz\" (UID: \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\") " pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:16.083563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:16.083533    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm" event={"ID":"6e5827f1-1262-4afe-be05-91c93daab648","Type":"ContainerStarted","Data":"3f2079dd76a8e39456173976197df6f9d961f084d25ef608bc73f1c94564bb96"}
Sep 10 21:16:16.085293 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:16.085269    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwkl\" (UniqueName: \"kubernetes.io/projected/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-kube-api-access-7xwkl\") pod \"payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz\" (UID: \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\") " pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:16.197304 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:16.197272    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:16.476176 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:16.476144    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"]
Sep 10 21:16:16.478481 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:16:16.478428    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4481ce4_cfd9_4b2f_850c_c73ccd78af51.slice/crio-16c20a55bf47283e8224e034f0acbfd2201e0d279154f5c51f849b46a61a13bc WatchSource:0}: Error finding container 16c20a55bf47283e8224e034f0acbfd2201e0d279154f5c51f849b46a61a13bc: Status 404 returned error can't find the container with id 16c20a55bf47283e8224e034f0acbfd2201e0d279154f5c51f849b46a61a13bc
Sep 10 21:16:17.093178 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:17.093108    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm" event={"ID":"6e5827f1-1262-4afe-be05-91c93daab648","Type":"ContainerStarted","Data":"15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118"}
Sep 10 21:16:17.093353 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:17.093309    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:17.094958 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:17.094924    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz" event={"ID":"f4481ce4-cfd9-4b2f-850c-c73ccd78af51","Type":"ContainerStarted","Data":"a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca"}
Sep 10 21:16:17.095142 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:17.094975    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz" event={"ID":"f4481ce4-cfd9-4b2f-850c-c73ccd78af51","Type":"ContainerStarted","Data":"16c20a55bf47283e8224e034f0acbfd2201e0d279154f5c51f849b46a61a13bc"}
Sep 10 21:16:17.095142 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:17.095045    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:17.114722 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:17.114669    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm" podStartSLOduration=1.5137394130000001 podStartE2EDuration="2.114655884s" totalImagesPullingTime="600.916471ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:17.111925131 +0000 UTC m=+1064.114171478" watchObservedRunningTime="2026-09-10 21:16:17.114655884 +0000 UTC m=+1064.116902235"
Sep 10 21:16:17.134921 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:17.134881    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz" podStartSLOduration=2.1348677990000002 podStartE2EDuration="2.134867799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:16:17.133460895 +0000 UTC m=+1064.135707244" watchObservedRunningTime="2026-09-10 21:16:17.134867799 +0000 UTC m=+1064.137114243"
Sep 10 21:16:20.712257 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:20.712221    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"]
Sep 10 21:16:20.713051 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:20.713016    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb" podUID="c577d9d4-9bee-47f3-82bd-1d5a68caba13" containerName="maas-api" containerID="cri-o://9904900eeed284f6dd395ae45b49614c51f46d2a8249cf4f691d68e52096bdef" gracePeriod=30
Sep 10 21:16:21.120977 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.120948    2580 generic.go:356] "Generic (PLEG): container finished" podID="c577d9d4-9bee-47f3-82bd-1d5a68caba13" containerID="9904900eeed284f6dd395ae45b49614c51f46d2a8249cf4f691d68e52096bdef" exitCode=0
Sep 10 21:16:21.121191 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.121016    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb" event={"ID":"c577d9d4-9bee-47f3-82bd-1d5a68caba13","Type":"ContainerDied","Data":"9904900eeed284f6dd395ae45b49614c51f46d2a8249cf4f691d68e52096bdef"}
Sep 10 21:16:21.227732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.227707    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:16:21.328391 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.328303    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-metrics-certs\") pod \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") "
Sep 10 21:16:21.328391 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.328361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvfzb\" (UniqueName: \"kubernetes.io/projected/c577d9d4-9bee-47f3-82bd-1d5a68caba13-kube-api-access-pvfzb\") pod \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") "
Sep 10 21:16:21.328391 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.328384    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-maas-api-tls\") pod \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") "
Sep 10 21:16:21.328661 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.328470    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c577d9d4-9bee-47f3-82bd-1d5a68caba13-openshift-service-ca\") pod \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\" (UID: \"c577d9d4-9bee-47f3-82bd-1d5a68caba13\") "
Sep 10 21:16:21.328812 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.328785    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c577d9d4-9bee-47f3-82bd-1d5a68caba13-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c577d9d4-9bee-47f3-82bd-1d5a68caba13" (UID: "c577d9d4-9bee-47f3-82bd-1d5a68caba13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:21.330567 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.330541    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c577d9d4-9bee-47f3-82bd-1d5a68caba13" (UID: "c577d9d4-9bee-47f3-82bd-1d5a68caba13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:21.330567 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.330557    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c577d9d4-9bee-47f3-82bd-1d5a68caba13" (UID: "c577d9d4-9bee-47f3-82bd-1d5a68caba13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:21.330713 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.330587    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c577d9d4-9bee-47f3-82bd-1d5a68caba13-kube-api-access-pvfzb" (OuterVolumeSpecName: "kube-api-access-pvfzb") pod "c577d9d4-9bee-47f3-82bd-1d5a68caba13" (UID: "c577d9d4-9bee-47f3-82bd-1d5a68caba13"). InnerVolumeSpecName "kube-api-access-pvfzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:21.429342 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.429307    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:21.429342 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.429335    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvfzb\" (UniqueName: \"kubernetes.io/projected/c577d9d4-9bee-47f3-82bd-1d5a68caba13-kube-api-access-pvfzb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:21.429342 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.429346    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c577d9d4-9bee-47f3-82bd-1d5a68caba13-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:21.429571 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:21.429355    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c577d9d4-9bee-47f3-82bd-1d5a68caba13-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:22.128250 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:22.128217    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"
Sep 10 21:16:22.128250 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:22.128244    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb" event={"ID":"c577d9d4-9bee-47f3-82bd-1d5a68caba13","Type":"ContainerDied","Data":"91d23a66c5efd79281bd7c0c33f97d29518af8a50f12d327887df604bbe241c4"}
Sep 10 21:16:22.128824 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:22.128306    2580 scope.go:117] "RemoveContainer" containerID="9904900eeed284f6dd395ae45b49614c51f46d2a8249cf4f691d68e52096bdef"
Sep 10 21:16:22.145622 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:22.145593    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"]
Sep 10 21:16:22.151888 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:22.151866    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-b22a78-7d58f647b9-p48zb"]
Sep 10 21:16:23.106470 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:23.106438    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:23.520852 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:23.520819    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c577d9d4-9bee-47f3-82bd-1d5a68caba13" path="/var/lib/kubelet/pods/c577d9d4-9bee-47f3-82bd-1d5a68caba13/volumes"
Sep 10 21:16:28.102822 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:28.102787    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:28.451239 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:28.451188    2580 patch_prober.go:28] interesting pod/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": read tcp 10.133.0.2:47548->10.133.0.69:15021: read: connection reset by peer" start-of-body=
Sep 10 21:16:28.451440 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:28.451273    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": read tcp 10.133.0.2:47548->10.133.0.69:15021: read: connection reset by peer"
Sep 10 21:16:29.170976 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:29.170945    2580 generic.go:356] "Generic (PLEG): container finished" podID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerID="4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865" exitCode=0
Sep 10 21:16:29.171436 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:29.171024    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" event={"ID":"7bca6d2e-05dd-48e4-9533-0587c5949e0a","Type":"ContainerDied","Data":"4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865"}
Sep 10 21:16:29.171511 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:29.171483    2580 scope.go:117] "RemoveContainer" containerID="4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865"
Sep 10 21:16:29.172585 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:29.172557    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:16:29.172671 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:29.172646    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:16:29.172708 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:29.172694    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:16:30.180721 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:30.180674    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" event={"ID":"7bca6d2e-05dd-48e4-9533-0587c5949e0a","Type":"ContainerStarted","Data":"9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242"}
Sep 10 21:16:31.105848 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:31.105808    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:31.106022 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:31.106004    2580 patch_prober.go:28] interesting pod/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 21:16:31.106077 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:31.106054    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 21:16:32.106344 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:32.106297    2580 patch_prober.go:28] interesting pod/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 21:16:32.106737 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:32.106378    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 21:16:33.106463 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:33.106423    2580 patch_prober.go:28] interesting pod/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 21:16:33.106814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:33.106501    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 21:16:34.106295 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:34.106256    2580 patch_prober.go:28] interesting pod/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 21:16:34.106517 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:34.106340    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 21:16:35.106231 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:35.106192    2580 patch_prober.go:28] interesting pod/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Sep 10 21:16:35.106428 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:35.106249    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Sep 10 21:16:36.109617 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:36.109589    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:36.109984 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:36.109858    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:36.110460 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:36.110439    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:16:38.874997 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:38.874960    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"]
Sep 10 21:16:38.875444 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:38.875165    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg" podUID="aa77cb97-c9cc-455f-913a-437ed371db23" containerName="maas-api" containerID="cri-o://104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d" gracePeriod=30
Sep 10 21:16:39.133154 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.133077    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:16:39.197958 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.197926    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa77cb97-c9cc-455f-913a-437ed371db23-openshift-service-ca\") pod \"aa77cb97-c9cc-455f-913a-437ed371db23\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") "
Sep 10 21:16:39.198133 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.198010    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xmhf\" (UniqueName: \"kubernetes.io/projected/aa77cb97-c9cc-455f-913a-437ed371db23-kube-api-access-5xmhf\") pod \"aa77cb97-c9cc-455f-913a-437ed371db23\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") "
Sep 10 21:16:39.198133 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.198032    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-metrics-certs\") pod \"aa77cb97-c9cc-455f-913a-437ed371db23\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") "
Sep 10 21:16:39.198133 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.198057    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls\") pod \"aa77cb97-c9cc-455f-913a-437ed371db23\" (UID: \"aa77cb97-c9cc-455f-913a-437ed371db23\") "
Sep 10 21:16:39.198320 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.198298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa77cb97-c9cc-455f-913a-437ed371db23-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa77cb97-c9cc-455f-913a-437ed371db23" (UID: "aa77cb97-c9cc-455f-913a-437ed371db23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:39.200301 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.200270    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aa77cb97-c9cc-455f-913a-437ed371db23" (UID: "aa77cb97-c9cc-455f-913a-437ed371db23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:39.200418 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.200343    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa77cb97-c9cc-455f-913a-437ed371db23-kube-api-access-5xmhf" (OuterVolumeSpecName: "kube-api-access-5xmhf") pod "aa77cb97-c9cc-455f-913a-437ed371db23" (UID: "aa77cb97-c9cc-455f-913a-437ed371db23"). InnerVolumeSpecName "kube-api-access-5xmhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:39.200418 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.200355    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa77cb97-c9cc-455f-913a-437ed371db23" (UID: "aa77cb97-c9cc-455f-913a-437ed371db23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:39.240365 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.240335    2580 generic.go:356] "Generic (PLEG): container finished" podID="aa77cb97-c9cc-455f-913a-437ed371db23" containerID="104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d" exitCode=0
Sep 10 21:16:39.240469 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.240383    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"
Sep 10 21:16:39.240469 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.240411    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg" event={"ID":"aa77cb97-c9cc-455f-913a-437ed371db23","Type":"ContainerDied","Data":"104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d"}
Sep 10 21:16:39.240469 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.240463    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg" event={"ID":"aa77cb97-c9cc-455f-913a-437ed371db23","Type":"ContainerDied","Data":"714bc9aa78a88c731bac2e563d1b300dda49623b7d1c317c5391c6b0dff57aca"}
Sep 10 21:16:39.240589 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.240483    2580 scope.go:117] "RemoveContainer" containerID="104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d"
Sep 10 21:16:39.251057 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.251037    2580 scope.go:117] "RemoveContainer" containerID="104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d"
Sep 10 21:16:39.251389 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:16:39.251369    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d\": container with ID starting with 104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d not found: ID does not exist" containerID="104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d"
Sep 10 21:16:39.251487 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.251396    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d"} err="failed to get container status \"104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d\": rpc error: code = NotFound desc = could not find container \"104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d\": container with ID starting with 104768b5f368b09af62272eed55f1a8466389ec36810fc6ee0fbf319ced6550d not found: ID does not exist"
Sep 10 21:16:39.266175 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.266150    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"]
Sep 10 21:16:39.271760 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.271741    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-414d5f-58cf9dbf9d-zvvbg"]
Sep 10 21:16:39.299513 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.299492    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xmhf\" (UniqueName: \"kubernetes.io/projected/aa77cb97-c9cc-455f-913a-437ed371db23-kube-api-access-5xmhf\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:39.299513 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.299511    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:39.299631 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.299522    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa77cb97-c9cc-455f-913a-437ed371db23-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:39.299631 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.299531    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa77cb97-c9cc-455f-913a-437ed371db23-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:39.519974 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:39.519894    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa77cb97-c9cc-455f-913a-437ed371db23" path="/var/lib/kubelet/pods/aa77cb97-c9cc-455f-913a-437ed371db23/volumes"
Sep 10 21:16:57.617020 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:57.616983    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"]
Sep 10 21:16:57.617601 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:57.617387    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm" podUID="6e5827f1-1262-4afe-be05-91c93daab648" containerName="maas-api" containerID="cri-o://15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118" gracePeriod=30
Sep 10 21:16:57.670354 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:57.670317    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"]
Sep 10 21:16:57.670983 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:57.670702    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz" podUID="f4481ce4-cfd9-4b2f-850c-c73ccd78af51" containerName="payload-processing" containerID="cri-o://a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca" gracePeriod=30
Sep 10 21:16:57.920018 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:57.919993    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:57.952632 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:57.952603    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:58.079549 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.079517    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-metrics-certs\") pod \"6e5827f1-1262-4afe-be05-91c93daab648\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") "
Sep 10 21:16:58.079549 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.079560    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-plugins-config-volume\") pod \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\" (UID: \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\") "
Sep 10 21:16:58.079805 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.079587    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xwkl\" (UniqueName: \"kubernetes.io/projected/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-kube-api-access-7xwkl\") pod \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\" (UID: \"f4481ce4-cfd9-4b2f-850c-c73ccd78af51\") "
Sep 10 21:16:58.079805 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.079644    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5827f1-1262-4afe-be05-91c93daab648-openshift-service-ca\") pod \"6e5827f1-1262-4afe-be05-91c93daab648\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") "
Sep 10 21:16:58.079805 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.079672    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2xvx\" (UniqueName: \"kubernetes.io/projected/6e5827f1-1262-4afe-be05-91c93daab648-kube-api-access-f2xvx\") pod \"6e5827f1-1262-4afe-be05-91c93daab648\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") "
Sep 10 21:16:58.079805 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.079699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-maas-api-tls\") pod \"6e5827f1-1262-4afe-be05-91c93daab648\" (UID: \"6e5827f1-1262-4afe-be05-91c93daab648\") "
Sep 10 21:16:58.080162 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.080011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f4481ce4-cfd9-4b2f-850c-c73ccd78af51" (UID: "f4481ce4-cfd9-4b2f-850c-c73ccd78af51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:58.080162 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.080046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e5827f1-1262-4afe-be05-91c93daab648-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e5827f1-1262-4afe-be05-91c93daab648" (UID: "6e5827f1-1262-4afe-be05-91c93daab648"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:16:58.082282 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.082248    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e5827f1-1262-4afe-be05-91c93daab648-kube-api-access-f2xvx" (OuterVolumeSpecName: "kube-api-access-f2xvx") pod "6e5827f1-1262-4afe-be05-91c93daab648" (UID: "6e5827f1-1262-4afe-be05-91c93daab648"). InnerVolumeSpecName "kube-api-access-f2xvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:58.082391 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.082370    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-kube-api-access-7xwkl" (OuterVolumeSpecName: "kube-api-access-7xwkl") pod "f4481ce4-cfd9-4b2f-850c-c73ccd78af51" (UID: "f4481ce4-cfd9-4b2f-850c-c73ccd78af51"). InnerVolumeSpecName "kube-api-access-7xwkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:16:58.082612 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.082591    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e5827f1-1262-4afe-be05-91c93daab648" (UID: "6e5827f1-1262-4afe-be05-91c93daab648"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:58.082671 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.082605    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e5827f1-1262-4afe-be05-91c93daab648" (UID: "6e5827f1-1262-4afe-be05-91c93daab648"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:16:58.180637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.180567    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:58.180637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.180599    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:58.180637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.180616    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xwkl\" (UniqueName: \"kubernetes.io/projected/f4481ce4-cfd9-4b2f-850c-c73ccd78af51-kube-api-access-7xwkl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:58.180637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.180630    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e5827f1-1262-4afe-be05-91c93daab648-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:58.180845 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.180643    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2xvx\" (UniqueName: \"kubernetes.io/projected/6e5827f1-1262-4afe-be05-91c93daab648-kube-api-access-f2xvx\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:58.180845 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.180656    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e5827f1-1262-4afe-be05-91c93daab648-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:16:58.348529 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.348494    2580 generic.go:356] "Generic (PLEG): container finished" podID="f4481ce4-cfd9-4b2f-850c-c73ccd78af51" containerID="a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca" exitCode=0
Sep 10 21:16:58.348684 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.348540    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"
Sep 10 21:16:58.348684 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.348579    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz" event={"ID":"f4481ce4-cfd9-4b2f-850c-c73ccd78af51","Type":"ContainerDied","Data":"a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca"}
Sep 10 21:16:58.348684 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.348625    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz" event={"ID":"f4481ce4-cfd9-4b2f-850c-c73ccd78af51","Type":"ContainerDied","Data":"16c20a55bf47283e8224e034f0acbfd2201e0d279154f5c51f849b46a61a13bc"}
Sep 10 21:16:58.348684 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.348647    2580 scope.go:117] "RemoveContainer" containerID="a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca"
Sep 10 21:16:58.350023 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.350005    2580 generic.go:356] "Generic (PLEG): container finished" podID="6e5827f1-1262-4afe-be05-91c93daab648" containerID="15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118" exitCode=0
Sep 10 21:16:58.350144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.350031    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm" event={"ID":"6e5827f1-1262-4afe-be05-91c93daab648","Type":"ContainerDied","Data":"15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118"}
Sep 10 21:16:58.350144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.350070    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm" event={"ID":"6e5827f1-1262-4afe-be05-91c93daab648","Type":"ContainerDied","Data":"3f2079dd76a8e39456173976197df6f9d961f084d25ef608bc73f1c94564bb96"}
Sep 10 21:16:58.350144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.350074    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"
Sep 10 21:16:58.357955 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.357916    2580 scope.go:117] "RemoveContainer" containerID="a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca"
Sep 10 21:16:58.358292 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:16:58.358271    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca\": container with ID starting with a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca not found: ID does not exist" containerID="a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca"
Sep 10 21:16:58.358372 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.358297    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca"} err="failed to get container status \"a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca\": rpc error: code = NotFound desc = could not find container \"a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca\": container with ID starting with a87fe0bdc8d776db179257f98d595ca9eaa3748ad8284894309fc36d327e43ca not found: ID does not exist"
Sep 10 21:16:58.358372 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.358313    2580 scope.go:117] "RemoveContainer" containerID="15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118"
Sep 10 21:16:58.366844 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.366828    2580 scope.go:117] "RemoveContainer" containerID="15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118"
Sep 10 21:16:58.367105 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:16:58.367089    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118\": container with ID starting with 15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118 not found: ID does not exist" containerID="15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118"
Sep 10 21:16:58.367177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.367111    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118"} err="failed to get container status \"15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118\": rpc error: code = NotFound desc = could not find container \"15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118\": container with ID starting with 15aa179e109c14ad1a9917ccef69140a3841754094dfbc8ac16d046b98a2d118 not found: ID does not exist"
Sep 10 21:16:58.374553 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.374531    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"]
Sep 10 21:16:58.378961 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.378941    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-170a718d-8656bc6596-vz4fm"]
Sep 10 21:16:58.392370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.392350    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"]
Sep 10 21:16:58.398569 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:58.398549    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-170a718d-68cf845b5c-tctvz"]
Sep 10 21:16:59.519135 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:59.519099    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e5827f1-1262-4afe-be05-91c93daab648" path="/var/lib/kubelet/pods/6e5827f1-1262-4afe-be05-91c93daab648/volumes"
Sep 10 21:16:59.519505 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:16:59.519457    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4481ce4-cfd9-4b2f-850c-c73ccd78af51" path="/var/lib/kubelet/pods/f4481ce4-cfd9-4b2f-850c-c73ccd78af51/volumes"
Sep 10 21:17:04.860148 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:04.857485    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"]
Sep 10 21:17:04.860148 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:04.857772    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy" containerID="cri-o://9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242" gracePeriod=30
Sep 10 21:17:06.112781 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:06.112746    2580 patch_prober.go:28] interesting pod/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 21:17:06.113172 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:06.112815    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 21:17:10.118144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.118097    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:17:10.185689 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185603    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zhxm\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-kube-api-access-2zhxm\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.185689 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185643    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-socket\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.185689 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185668    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-podinfo\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.185963 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185719    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-data\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.185963 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185742    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-certs\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.185963 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185769    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-envoy\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.185963 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185880    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istiod-ca-cert\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.185963 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185905    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-credential-socket\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.185963 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185933    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-token\") pod \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\" (UID: \"7bca6d2e-05dd-48e4-9533-0587c5949e0a\") "
Sep 10 21:17:10.186308 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.185975    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:17:10.186308 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.186049    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:17:10.186308 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.186070    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:17:10.186308 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.186259    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.186308 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.186265    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:17:10.186308 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.186277    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.186308 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.186291    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.186852 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.186352    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:17:10.188081 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.188056    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:17:10.188231 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.188080    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-kube-api-access-2zhxm" (OuterVolumeSpecName: "kube-api-access-2zhxm") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "kube-api-access-2zhxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:10.188284 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.188231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:17:10.188319 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.188279    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bca6d2e-05dd-48e4-9533-0587c5949e0a" (UID: "7bca6d2e-05dd-48e4-9533-0587c5949e0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:17:10.286960 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.286921    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.286960 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.286944    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.286960 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.286953    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.286960 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.286962    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zhxm\" (UniqueName: \"kubernetes.io/projected/7bca6d2e-05dd-48e4-9533-0587c5949e0a-kube-api-access-2zhxm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.287313 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.286971    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.287313 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.286979    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bca6d2e-05dd-48e4-9533-0587c5949e0a-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:17:10.418969 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.418929    2580 generic.go:356] "Generic (PLEG): container finished" podID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerID="9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242" exitCode=0
Sep 10 21:17:10.419145 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.418983    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"
Sep 10 21:17:10.419145 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.418990    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" event={"ID":"7bca6d2e-05dd-48e4-9533-0587c5949e0a","Type":"ContainerDied","Data":"9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242"}
Sep 10 21:17:10.419145 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.419045    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4" event={"ID":"7bca6d2e-05dd-48e4-9533-0587c5949e0a","Type":"ContainerDied","Data":"b8394121f90083c6a67220770d94c151e6da046209693413b518bc5bcb1f8f0d"}
Sep 10 21:17:10.419145 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.419075    2580 scope.go:117] "RemoveContainer" containerID="9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242"
Sep 10 21:17:10.428864 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.428844    2580 scope.go:117] "RemoveContainer" containerID="4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865"
Sep 10 21:17:10.437282 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.437265    2580 scope.go:117] "RemoveContainer" containerID="9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242"
Sep 10 21:17:10.437561 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:17:10.437540    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242\": container with ID starting with 9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242 not found: ID does not exist" containerID="9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242"
Sep 10 21:17:10.437604 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.437569    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242"} err="failed to get container status \"9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242\": rpc error: code = NotFound desc = could not find container \"9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242\": container with ID starting with 9cc3b644a87d30e72962e91eed71dd66eadbb9484cc53d91cb86859033369242 not found: ID does not exist"
Sep 10 21:17:10.437604 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.437587    2580 scope.go:117] "RemoveContainer" containerID="4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865"
Sep 10 21:17:10.437824 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:17:10.437808    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865\": container with ID starting with 4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865 not found: ID does not exist" containerID="4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865"
Sep 10 21:17:10.437872 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.437830    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865"} err="failed to get container status \"4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865\": rpc error: code = NotFound desc = could not find container \"4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865\": container with ID starting with 4f47fbdbdcb77777c65a5585f130d6b87dffc3df58c9afb1f55e6b2188743865 not found: ID does not exist"
Sep 10 21:17:10.441560 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.441540    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"]
Sep 10 21:17:10.445879 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:10.445858    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-170a718d-openshift-default-84789669b4-fqfj4"]
Sep 10 21:17:11.519384 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:11.519349    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" path="/var/lib/kubelet/pods/7bca6d2e-05dd-48e4-9533-0587c5949e0a/volumes"
Sep 10 21:17:15.409682 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.409652    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"]
Sep 10 21:17:15.410171 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410145    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c577d9d4-9bee-47f3-82bd-1d5a68caba13" containerName="maas-api"
Sep 10 21:17:15.410171 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410161    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c577d9d4-9bee-47f3-82bd-1d5a68caba13" containerName="maas-api"
Sep 10 21:17:15.410171 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410171    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa77cb97-c9cc-455f-913a-437ed371db23" containerName="maas-api"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410176    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa77cb97-c9cc-455f-913a-437ed371db23" containerName="maas-api"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410183    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410189    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410201    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4481ce4-cfd9-4b2f-850c-c73ccd78af51" containerName="payload-processing"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410206    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4481ce4-cfd9-4b2f-850c-c73ccd78af51" containerName="payload-processing"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410216    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e5827f1-1262-4afe-be05-91c93daab648" containerName="maas-api"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410221    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e5827f1-1262-4afe-be05-91c93daab648" containerName="maas-api"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410233    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy"
Sep 10 21:17:15.410290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410238    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy"
Sep 10 21:17:15.410605 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410299    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa77cb97-c9cc-455f-913a-437ed371db23" containerName="maas-api"
Sep 10 21:17:15.410605 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410306    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy"
Sep 10 21:17:15.410605 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410315    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c577d9d4-9bee-47f3-82bd-1d5a68caba13" containerName="maas-api"
Sep 10 21:17:15.410605 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410320    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e5827f1-1262-4afe-be05-91c93daab648" containerName="maas-api"
Sep 10 21:17:15.410605 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.410326    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4481ce4-cfd9-4b2f-850c-c73ccd78af51" containerName="payload-processing"
Sep 10 21:17:15.414788 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.414763    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.416905 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.416882    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 21:17:15.416905 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.416908    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-16bbec-openshift-default-dockercfg-8qzm5\""
Sep 10 21:17:15.425665 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.425639    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"]
Sep 10 21:17:15.532921 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.532891    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-podinfo\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.532921 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.532923    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-token\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.533177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.532944    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-certs\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.533177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.532960    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-envoy\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.533177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.533013    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-data\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.533177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.533046    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs5pg\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-kube-api-access-cs5pg\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.533177 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.533141    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497ce30-4b56-410a-967f-4b09b7629e8f-istiod-ca-cert\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.533406 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.533195    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-socket\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.533406 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.533258    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-credential-socket\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634269 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634225    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-credential-socket\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634269 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634276    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-podinfo\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634297    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-token\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634316    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-certs\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634331    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-envoy\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634453    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-data\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634522 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634499    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs5pg\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-kube-api-access-cs5pg\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634781 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634564    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497ce30-4b56-410a-967f-4b09b7629e8f-istiod-ca-cert\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634781 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634681    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-credential-socket\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634781 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634695    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-certs\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634939 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634792    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-data\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.634939 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.634796    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-socket\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.635042 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.635024    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-socket\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.635333 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.635314    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497ce30-4b56-410a-967f-4b09b7629e8f-istiod-ca-cert\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.636785 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.636762    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-envoy\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.637158 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.637138    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-podinfo\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.642634 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.642608    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-token\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.642951 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.642930    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs5pg\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-kube-api-access-cs5pg\") pod \"e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") " pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:15.726074 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:15.726004    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:16.005141 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:16.005097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"]
Sep 10 21:17:16.006635 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:17:16.006607    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8497ce30_4b56_410a_967f_4b09b7629e8f.slice/crio-72b4108742c1367e1e57bad78ab333827efdbd8d6da8d6bc4a43aed84091664c WatchSource:0}: Error finding container 72b4108742c1367e1e57bad78ab333827efdbd8d6da8d6bc4a43aed84091664c: Status 404 returned error can't find the container with id 72b4108742c1367e1e57bad78ab333827efdbd8d6da8d6bc4a43aed84091664c
Sep 10 21:17:16.008898 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:16.008865    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:16.008985 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:16.008933    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:16.008985 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:16.008957    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:16.453732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:16.453698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" event={"ID":"8497ce30-4b56-410a-967f-4b09b7629e8f","Type":"ContainerStarted","Data":"02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37"}
Sep 10 21:17:16.454089 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:16.453758    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" event={"ID":"8497ce30-4b56-410a-967f-4b09b7629e8f","Type":"ContainerStarted","Data":"72b4108742c1367e1e57bad78ab333827efdbd8d6da8d6bc4a43aed84091664c"}
Sep 10 21:17:16.476750 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:16.476696    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podStartSLOduration=1.476679447 podStartE2EDuration="1.476679447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:16.475075478 +0000 UTC m=+1123.477321825" watchObservedRunningTime="2026-09-10 21:17:16.476679447 +0000 UTC m=+1123.478925794"
Sep 10 21:17:16.726287 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:16.726195    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:17.731782 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:17.731753    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:18.464478 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:18.464449    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:18.465455 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:18.465431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:19.524398 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.524371    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"]
Sep 10 21:17:19.524785 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.524771    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bca6d2e-05dd-48e4-9533-0587c5949e0a" containerName="istio-proxy"
Sep 10 21:17:19.527801 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.527787    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.530002 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.529979    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-501c57-openshift-default-dockercfg-gwlrq\""
Sep 10 21:17:19.541840 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.541815    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"]
Sep 10 21:17:19.570897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.570866    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-credential-socket\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.571061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.570920    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-envoy\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.571061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.570947    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-data\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.571061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.570972    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzb8z\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-kube-api-access-dzb8z\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.571298 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.571060    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-socket\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.571298 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.571091    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-token\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.571298 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.571223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed59769b-454d-4358-b0b2-2334cd4542a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.571298 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.571267    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-podinfo\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.571519 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.571315    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-certs\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.620470 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.620436    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"]
Sep 10 21:17:19.623814 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.623796    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.626312 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.626294    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-16bbec-serving-cert\""
Sep 10 21:17:19.635725 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.635706    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"]
Sep 10 21:17:19.672109 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672071    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.672267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672115    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed59769b-454d-4358-b0b2-2334cd4542a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672185    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-podinfo\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672217    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.672267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672246    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-certs\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672487 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672356    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-credential-socket\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672487 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672393    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-envoy\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672487 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672417    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-data\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672487 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzb8z\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-kube-api-access-dzb8z\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672693 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672554    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnj6r\" (UniqueName: \"kubernetes.io/projected/22a941b7-dc79-42d2-b4b5-e3249453a50c-kube-api-access-pnj6r\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.672693 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672614    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-socket\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672693 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672642    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-token\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672693 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672682    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a941b7-dc79-42d2-b4b5-e3249453a50c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.672896 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672771    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-data\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672896 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672807    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-credential-socket\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672896 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672861    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed59769b-454d-4358-b0b2-2334cd4542a4-istiod-ca-cert\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.672896 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672884    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-socket\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.673306 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.672961    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-certs\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.674736 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.674716    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-envoy\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.674937 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.674921    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-podinfo\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.680251 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.680224    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzb8z\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-kube-api-access-dzb8z\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.680334 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.680262    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-token\") pod \"e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") " pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:19.773199 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.773166    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.773386 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.773268    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnj6r\" (UniqueName: \"kubernetes.io/projected/22a941b7-dc79-42d2-b4b5-e3249453a50c-kube-api-access-pnj6r\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.773386 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.773311    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a941b7-dc79-42d2-b4b5-e3249453a50c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.773386 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.773348    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.773565 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:17:19.773450    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-serving-cert: secret "maas-api-e2e-shared-b-w3-16bbec-serving-cert" not found
Sep 10 21:17:19.773565 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:17:19.773521    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls podName:22a941b7-dc79-42d2-b4b5-e3249453a50c nodeName:}" failed. No retries permitted until 2026-09-10 21:17:20.273499016 +0000 UTC m=+1127.275745347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls") pod "maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz" (UID: "22a941b7-dc79-42d2-b4b5-e3249453a50c") : secret "maas-api-e2e-shared-b-w3-16bbec-serving-cert" not found
Sep 10 21:17:19.774002 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.773982    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a941b7-dc79-42d2-b4b5-e3249453a50c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.775802 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.775754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.782576 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.782553    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnj6r\" (UniqueName: \"kubernetes.io/projected/22a941b7-dc79-42d2-b4b5-e3249453a50c-kube-api-access-pnj6r\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:19.838460 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:19.838430    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:20.132931 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.132901    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"]
Sep 10 21:17:20.136463 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:17:20.136412    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded59769b_454d_4358_b0b2_2334cd4542a4.slice/crio-f17992b643ce23f129df9b5f503a9f35f3f2421c9dcf235158bc8b0fdc13dbd3 WatchSource:0}: Error finding container f17992b643ce23f129df9b5f503a9f35f3f2421c9dcf235158bc8b0fdc13dbd3: Status 404 returned error can't find the container with id f17992b643ce23f129df9b5f503a9f35f3f2421c9dcf235158bc8b0fdc13dbd3
Sep 10 21:17:20.139666 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.139638    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:20.139732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.139716    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:20.139774 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.139746    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:20.277770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.277732    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:20.277957 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:17:20.277912    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-serving-cert: secret "maas-api-e2e-shared-b-w3-16bbec-serving-cert" not found
Sep 10 21:17:20.278023 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:17:20.277994    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls podName:22a941b7-dc79-42d2-b4b5-e3249453a50c nodeName:}" failed. No retries permitted until 2026-09-10 21:17:21.27797303 +0000 UTC m=+1128.280219373 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls") pod "maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz" (UID: "22a941b7-dc79-42d2-b4b5-e3249453a50c") : secret "maas-api-e2e-shared-b-w3-16bbec-serving-cert" not found
Sep 10 21:17:20.425775 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.425701    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"]
Sep 10 21:17:20.429216 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.429191    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:20.431543 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.431522    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-16bbec-dockercfg-ttcd7\""
Sep 10 21:17:20.431629 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.431522    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-16bbec\""
Sep 10 21:17:20.440955 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.440936    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"]
Sep 10 21:17:20.479205 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.479165    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" event={"ID":"ed59769b-454d-4358-b0b2-2334cd4542a4","Type":"ContainerStarted","Data":"0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7"}
Sep 10 21:17:20.479372 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.479238    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" event={"ID":"ed59769b-454d-4358-b0b2-2334cd4542a4","Type":"ContainerStarted","Data":"f17992b643ce23f129df9b5f503a9f35f3f2421c9dcf235158bc8b0fdc13dbd3"}
Sep 10 21:17:20.479947 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.479920    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn\" (UID: \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:20.480090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.479987    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ptvp\" (UniqueName: \"kubernetes.io/projected/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-kube-api-access-5ptvp\") pod \"payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn\" (UID: \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:20.500946 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.500886    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" podStartSLOduration=1.50086803 podStartE2EDuration="1.50086803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:20.498213889 +0000 UTC m=+1127.500460233" watchObservedRunningTime="2026-09-10 21:17:20.50086803 +0000 UTC m=+1127.503114376"
Sep 10 21:17:20.580925 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.580885    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn\" (UID: \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:20.581420 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.581256    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ptvp\" (UniqueName: \"kubernetes.io/projected/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-kube-api-access-5ptvp\") pod \"payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn\" (UID: \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:20.581831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.581805    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn\" (UID: \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:20.590115 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.590094    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ptvp\" (UniqueName: \"kubernetes.io/projected/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-kube-api-access-5ptvp\") pod \"payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn\" (UID: \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:20.741506 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.741427    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:20.838708 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:20.838683    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:21.017624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.017599    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"]
Sep 10 21:17:21.019273 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:17:21.019247    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c41916_3e9a_4bc0_99b7_d13a45c062a7.slice/crio-6784efb5250f7c840acd88f7ae17f5deef7801a9052e7d788fcfe477065e685e WatchSource:0}: Error finding container 6784efb5250f7c840acd88f7ae17f5deef7801a9052e7d788fcfe477065e685e: Status 404 returned error can't find the container with id 6784efb5250f7c840acd88f7ae17f5deef7801a9052e7d788fcfe477065e685e
Sep 10 21:17:21.288535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.288443    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:21.291065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.291042    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:21.435036 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.435008    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:21.490295 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.489188    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn" event={"ID":"e6c41916-3e9a-4bc0-99b7-d13a45c062a7","Type":"ContainerStarted","Data":"cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a"}
Sep 10 21:17:21.490295 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.489243    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn" event={"ID":"e6c41916-3e9a-4bc0-99b7-d13a45c062a7","Type":"ContainerStarted","Data":"6784efb5250f7c840acd88f7ae17f5deef7801a9052e7d788fcfe477065e685e"}
Sep 10 21:17:21.490295 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.489442    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:21.510927 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.510882    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn" podStartSLOduration=1.510865344 podStartE2EDuration="1.510865344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:21.508682085 +0000 UTC m=+1128.510928432" watchObservedRunningTime="2026-09-10 21:17:21.510865344 +0000 UTC m=+1128.513111744"
Sep 10 21:17:21.710447 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.710411    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"]
Sep 10 21:17:21.712427 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:17:21.712398    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a941b7_dc79_42d2_b4b5_e3249453a50c.slice/crio-afe0506a33c2cc2848e60c57a5c6cd871062b2f944ca4f28c6fd9d5c1de16c34 WatchSource:0}: Error finding container afe0506a33c2cc2848e60c57a5c6cd871062b2f944ca4f28c6fd9d5c1de16c34: Status 404 returned error can't find the container with id afe0506a33c2cc2848e60c57a5c6cd871062b2f944ca4f28c6fd9d5c1de16c34
Sep 10 21:17:21.844153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:21.844093    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:22.497696 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:22.497599    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz" event={"ID":"22a941b7-dc79-42d2-b4b5-e3249453a50c","Type":"ContainerStarted","Data":"cfc2f35b16d3abd50acd864aec636f2893a37f537599dcb94c765cd1d9dc52de"}
Sep 10 21:17:22.497696 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:22.497661    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz" event={"ID":"22a941b7-dc79-42d2-b4b5-e3249453a50c","Type":"ContainerStarted","Data":"afe0506a33c2cc2848e60c57a5c6cd871062b2f944ca4f28c6fd9d5c1de16c34"}
Sep 10 21:17:22.497931 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:22.497851    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:22.498985 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:22.498961    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:22.518107 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:22.518042    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz" podStartSLOduration=3.102215832 podStartE2EDuration="3.518010285s" totalImagesPullingTime="415.794453ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:22.515215861 +0000 UTC m=+1129.517462209" watchObservedRunningTime="2026-09-10 21:17:22.518010285 +0000 UTC m=+1129.520256633"
Sep 10 21:17:23.504883 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:23.504845    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:24.340322 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.340288    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"]
Sep 10 21:17:24.343810 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.343795    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:24.345897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.345871    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-501c57\""
Sep 10 21:17:24.346032 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.345931    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-501c57-dockercfg-xf854\""
Sep 10 21:17:24.356413 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.356381    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"]
Sep 10 21:17:24.416555 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.416500    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgp6t\" (UniqueName: \"kubernetes.io/projected/5d360019-bff8-47e6-994f-ab1b1800dc64-kube-api-access-bgp6t\") pod \"payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj\" (UID: \"5d360019-bff8-47e6-994f-ab1b1800dc64\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:24.416769 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.416581    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d360019-bff8-47e6-994f-ab1b1800dc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj\" (UID: \"5d360019-bff8-47e6-994f-ab1b1800dc64\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:24.517788 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.517756    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgp6t\" (UniqueName: \"kubernetes.io/projected/5d360019-bff8-47e6-994f-ab1b1800dc64-kube-api-access-bgp6t\") pod \"payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj\" (UID: \"5d360019-bff8-47e6-994f-ab1b1800dc64\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:24.518277 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.517802    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d360019-bff8-47e6-994f-ab1b1800dc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj\" (UID: \"5d360019-bff8-47e6-994f-ab1b1800dc64\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:24.518505 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.518483    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d360019-bff8-47e6-994f-ab1b1800dc64-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj\" (UID: \"5d360019-bff8-47e6-994f-ab1b1800dc64\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:24.526658 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.526630    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgp6t\" (UniqueName: \"kubernetes.io/projected/5d360019-bff8-47e6-994f-ab1b1800dc64-kube-api-access-bgp6t\") pod \"payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj\" (UID: \"5d360019-bff8-47e6-994f-ab1b1800dc64\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:24.657235 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.657189    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:24.941732 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:24.941703    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"]
Sep 10 21:17:24.942706 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:17:24.942685    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d360019_bff8_47e6_994f_ab1b1800dc64.slice/crio-3d841edc61f7f7bea1bbd1dbbbb86cb33f11e5d9a70071f310a00b391ded4e5b WatchSource:0}: Error finding container 3d841edc61f7f7bea1bbd1dbbbb86cb33f11e5d9a70071f310a00b391ded4e5b: Status 404 returned error can't find the container with id 3d841edc61f7f7bea1bbd1dbbbb86cb33f11e5d9a70071f310a00b391ded4e5b
Sep 10 21:17:25.521597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:25.521561    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj" event={"ID":"5d360019-bff8-47e6-994f-ab1b1800dc64","Type":"ContainerStarted","Data":"d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105"}
Sep 10 21:17:25.522982 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:25.522962    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:25.523207 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:25.523191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj" event={"ID":"5d360019-bff8-47e6-994f-ab1b1800dc64","Type":"ContainerStarted","Data":"3d841edc61f7f7bea1bbd1dbbbb86cb33f11e5d9a70071f310a00b391ded4e5b"}
Sep 10 21:17:25.540024 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:25.539944    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj" podStartSLOduration=1.5399246469999999 podStartE2EDuration="1.539924647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:25.539079158 +0000 UTC m=+1132.541325554" watchObservedRunningTime="2026-09-10 21:17:25.539924647 +0000 UTC m=+1132.542170994"
Sep 10 21:17:29.525737 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:29.525711    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:17:32.500592 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:32.500553    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:17:34.153156 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:34.153096    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:40460->10.133.0.72:15021: read: connection reset by peer" start-of-body=
Sep 10 21:17:34.153486 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:34.153180    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": read tcp 10.133.0.2:40460->10.133.0.72:15021: read: connection reset by peer"
Sep 10 21:17:34.584438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:34.582592    2580 generic.go:356] "Generic (PLEG): container finished" podID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerID="02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37" exitCode=0
Sep 10 21:17:34.584438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:34.582744    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" event={"ID":"8497ce30-4b56-410a-967f-4b09b7629e8f","Type":"ContainerDied","Data":"02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37"}
Sep 10 21:17:34.584438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:34.583252    2580 scope.go:117] "RemoveContainer" containerID="02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37"
Sep 10 21:17:34.585375 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:34.585336    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:34.585686 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:34.585665    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:34.585873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:34.585844    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:35.623147 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:35.622655    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" event={"ID":"8497ce30-4b56-410a-967f-4b09b7629e8f","Type":"ContainerStarted","Data":"dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04"}
Sep 10 21:17:35.727147 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:35.727078    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:35.727503 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:35.727469    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 10 21:17:35.727571 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:35.727537    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 10 21:17:36.728145 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:36.727582    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 10 21:17:36.728145 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:36.727645    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 10 21:17:37.537183 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:37.537147    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:17:37.727289 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:37.727257    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 10 21:17:37.727529 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:37.727508    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 10 21:17:38.726602 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:38.726564    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 10 21:17:38.726980 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:38.726626    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 10 21:17:39.727100 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:39.727058    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 10 21:17:39.727594 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:39.727132    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 10 21:17:40.726969 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:40.726929    2580 patch_prober.go:28] interesting pod/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Sep 10 21:17:40.727178 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:40.726981    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Sep 10 21:17:41.672895 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.672865    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerID="0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7" exitCode=0
Sep 10 21:17:41.673084 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.672910    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" event={"ID":"ed59769b-454d-4358-b0b2-2334cd4542a4","Type":"ContainerDied","Data":"0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7"}
Sep 10 21:17:41.673355 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.673331    2580 scope.go:117] "RemoveContainer" containerID="0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7"
Sep 10 21:17:41.674398 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.674373    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:41.674491 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.674449    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:41.674491 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.674486    2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892140Ki","pods":"250"}
Sep 10 21:17:41.730849 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.730820    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:41.731411 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.731389    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:41.732202 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:41.732178    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:17:42.681406 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:42.681369    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" event={"ID":"ed59769b-454d-4358-b0b2-2334cd4542a4","Type":"ContainerStarted","Data":"323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7"}
Sep 10 21:17:42.839438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:42.839401    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:42.839822 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:42.839612    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:17:42.839822 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:42.839669    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 21:17:43.839831 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:43.839796    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:17:43.840399 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:43.839856    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 21:17:44.839474 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:44.839442    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:17:44.839652 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:44.839506    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 21:17:45.839730 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:45.839696    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:17:45.840101 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:45.839766    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 21:17:46.838929 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:46.838892    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:17:46.839153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:46.838954    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 21:17:47.838836 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:47.838803    2580 patch_prober.go:28] interesting pod/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Sep 10 21:17:47.839236 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:47.838857    2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Sep 10 21:17:48.843704 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:48.843674    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:48.844077 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:48.844053    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:48.844857 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:48.844840    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:17:52.355416 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.355376    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"]
Sep 10 21:17:52.358083 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.358061    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.360201 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.360170    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-4ad710-serving-cert\""
Sep 10 21:17:52.372135 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.372097    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"]
Sep 10 21:17:52.388645 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.388618    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4ckc\" (UniqueName: \"kubernetes.io/projected/10ad7576-2d43-4205-91dc-a98639086062-kube-api-access-r4ckc\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.388772 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.388663    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ad7576-2d43-4205-91dc-a98639086062-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.388772 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.388702    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.388772 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.388751    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.490276 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.490241    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.490453 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.490351    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4ckc\" (UniqueName: \"kubernetes.io/projected/10ad7576-2d43-4205-91dc-a98639086062-kube-api-access-r4ckc\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.490453 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.490395    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ad7576-2d43-4205-91dc-a98639086062-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.490453 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:17:52.490404    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-serving-cert: secret "maas-api-e2e-shared-b-w2-4ad710-serving-cert" not found
Sep 10 21:17:52.490453 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.490430    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.490691 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:17:52.490489    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls podName:10ad7576-2d43-4205-91dc-a98639086062 nodeName:}" failed. No retries permitted until 2026-09-10 21:17:52.990467841 +0000 UTC m=+1159.992714171 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls") pod "maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9" (UID: "10ad7576-2d43-4205-91dc-a98639086062") : secret "maas-api-e2e-shared-b-w2-4ad710-serving-cert" not found
Sep 10 21:17:52.491080 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.491055    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ad7576-2d43-4205-91dc-a98639086062-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.492952 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.492931    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.499216 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.499193    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4ckc\" (UniqueName: \"kubernetes.io/projected/10ad7576-2d43-4205-91dc-a98639086062-kube-api-access-r4ckc\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.996830 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.996781    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:52.999586 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:52.999563    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:53.005412 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.005391    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"]
Sep 10 21:17:53.008073 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.008057    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.010664 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.010641    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-4ad710-dockercfg-fz896\""
Sep 10 21:17:53.010778 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.010695    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-4ad710\""
Sep 10 21:17:53.018965 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.018944    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"]
Sep 10 21:17:53.098270 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.098240    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f033e79-94fa-4e8b-ae82-1fb32f893121-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4\" (UID: \"9f033e79-94fa-4e8b-ae82-1fb32f893121\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.098448 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.098303    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957r5\" (UniqueName: \"kubernetes.io/projected/9f033e79-94fa-4e8b-ae82-1fb32f893121-kube-api-access-957r5\") pod \"payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4\" (UID: \"9f033e79-94fa-4e8b-ae82-1fb32f893121\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.199636 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.199597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f033e79-94fa-4e8b-ae82-1fb32f893121-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4\" (UID: \"9f033e79-94fa-4e8b-ae82-1fb32f893121\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.199825 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.199655    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957r5\" (UniqueName: \"kubernetes.io/projected/9f033e79-94fa-4e8b-ae82-1fb32f893121-kube-api-access-957r5\") pod \"payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4\" (UID: \"9f033e79-94fa-4e8b-ae82-1fb32f893121\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.200318 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.200299    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f033e79-94fa-4e8b-ae82-1fb32f893121-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4\" (UID: \"9f033e79-94fa-4e8b-ae82-1fb32f893121\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.209211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.209190    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-957r5\" (UniqueName: \"kubernetes.io/projected/9f033e79-94fa-4e8b-ae82-1fb32f893121-kube-api-access-957r5\") pod \"payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4\" (UID: \"9f033e79-94fa-4e8b-ae82-1fb32f893121\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.269790 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.269712    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:53.324562 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.324538    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.548547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.548519    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"]
Sep 10 21:17:53.549984 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:17:53.549951    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ad7576_2d43_4205_91dc_a98639086062.slice/crio-57234be7e7891dfb329197e4942b20590f82b3de162e674a78bfe970b3c61195 WatchSource:0}: Error finding container 57234be7e7891dfb329197e4942b20590f82b3de162e674a78bfe970b3c61195: Status 404 returned error can't find the container with id 57234be7e7891dfb329197e4942b20590f82b3de162e674a78bfe970b3c61195
Sep 10 21:17:53.609900 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.609865    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"]
Sep 10 21:17:53.756013 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.755972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4" event={"ID":"9f033e79-94fa-4e8b-ae82-1fb32f893121","Type":"ContainerStarted","Data":"fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5"}
Sep 10 21:17:53.756252 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.756039    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4" event={"ID":"9f033e79-94fa-4e8b-ae82-1fb32f893121","Type":"ContainerStarted","Data":"e9972bf63bed6badb9cb8afe3f765fb74dfed932b0a9b59721ace74de7380b5e"}
Sep 10 21:17:53.756252 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.756090    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:17:53.757571 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.757546    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9" event={"ID":"10ad7576-2d43-4205-91dc-a98639086062","Type":"ContainerStarted","Data":"57234be7e7891dfb329197e4942b20590f82b3de162e674a78bfe970b3c61195"}
Sep 10 21:17:53.784140 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:53.784054    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4" podStartSLOduration=1.784033087 podStartE2EDuration="1.784033087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:53.780698993 +0000 UTC m=+1160.782945342" watchObservedRunningTime="2026-09-10 21:17:53.784033087 +0000 UTC m=+1160.786279436"
Sep 10 21:17:54.764997 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:54.764897    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9" event={"ID":"10ad7576-2d43-4205-91dc-a98639086062","Type":"ContainerStarted","Data":"935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e"}
Sep 10 21:17:54.765452 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:54.765156    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:17:54.784525 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:17:54.784479    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9" podStartSLOduration=2.372634211 podStartE2EDuration="2.784463002s" totalImagesPullingTime="411.828791ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:17:54.78229832 +0000 UTC m=+1161.784544667" watchObservedRunningTime="2026-09-10 21:17:54.784463002 +0000 UTC m=+1161.786709349"
Sep 10 21:18:00.776819 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:00.776790    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:18:04.767372 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:04.767341    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:18:32.348897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.348861    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"]
Sep 10 21:18:32.351986 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.351965    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:32.354184 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.354165    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7f376d02\""
Sep 10 21:18:32.369227 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.368839    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"]
Sep 10 21:18:32.485174 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.485115    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx\" (UID: \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:32.485470 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.485445    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdp2r\" (UniqueName: \"kubernetes.io/projected/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-kube-api-access-qdp2r\") pod \"payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx\" (UID: \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:32.586060 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.586016    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx\" (UID: \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:32.586060 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.586069    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdp2r\" (UniqueName: \"kubernetes.io/projected/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-kube-api-access-qdp2r\") pod \"payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx\" (UID: \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:32.586773 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.586747    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx\" (UID: \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:32.593779 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.593754    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdp2r\" (UniqueName: \"kubernetes.io/projected/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-kube-api-access-qdp2r\") pod \"payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx\" (UID: \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:32.662684 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.662660    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:32.938683 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.938652    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"]
Sep 10 21:18:32.941507 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:18:32.941477    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc502fa75_66c6_45a3_ad9f_7d0b28b3910f.slice/crio-75d4c2bb9671ec0d7baa2cc23c0c6271cb941037e12c7970f62c26fa60da855d WatchSource:0}: Error finding container 75d4c2bb9671ec0d7baa2cc23c0c6271cb941037e12c7970f62c26fa60da855d: Status 404 returned error can't find the container with id 75d4c2bb9671ec0d7baa2cc23c0c6271cb941037e12c7970f62c26fa60da855d
Sep 10 21:18:32.981779 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:32.981693    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx" event={"ID":"c502fa75-66c6-45a3-ad9f-7d0b28b3910f","Type":"ContainerStarted","Data":"75d4c2bb9671ec0d7baa2cc23c0c6271cb941037e12c7970f62c26fa60da855d"}
Sep 10 21:18:33.989900 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:33.989861    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx" event={"ID":"c502fa75-66c6-45a3-ad9f-7d0b28b3910f","Type":"ContainerStarted","Data":"4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f"}
Sep 10 21:18:33.990299 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:33.990041    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:18:34.008712 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:34.008662    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx" podStartSLOduration=2.008647732 podStartE2EDuration="2.008647732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:18:34.005920794 +0000 UTC m=+1201.008167150" watchObservedRunningTime="2026-09-10 21:18:34.008647732 +0000 UTC m=+1201.010894079"
Sep 10 21:18:44.998023 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:18:44.997989    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:19:55.859622 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:55.859542    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"]
Sep 10 21:19:55.860187 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:55.859810    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx" podUID="c502fa75-66c6-45a3-ad9f-7d0b28b3910f" containerName="payload-pre-processing" containerID="cri-o://4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f" gracePeriod=30
Sep 10 21:19:56.117903 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.117834    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:19:56.146331 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.146307    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdp2r\" (UniqueName: \"kubernetes.io/projected/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-kube-api-access-qdp2r\") pod \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\" (UID: \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\") "
Sep 10 21:19:56.146493 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.146361    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-plugins-config-volume\") pod \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\" (UID: \"c502fa75-66c6-45a3-ad9f-7d0b28b3910f\") "
Sep 10 21:19:56.146803 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.146765    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c502fa75-66c6-45a3-ad9f-7d0b28b3910f" (UID: "c502fa75-66c6-45a3-ad9f-7d0b28b3910f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:19:56.148934 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.148905    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-kube-api-access-qdp2r" (OuterVolumeSpecName: "kube-api-access-qdp2r") pod "c502fa75-66c6-45a3-ad9f-7d0b28b3910f" (UID: "c502fa75-66c6-45a3-ad9f-7d0b28b3910f"). InnerVolumeSpecName "kube-api-access-qdp2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:19:56.248104 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.248065    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdp2r\" (UniqueName: \"kubernetes.io/projected/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-kube-api-access-qdp2r\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:19:56.248104 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.248096    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c502fa75-66c6-45a3-ad9f-7d0b28b3910f-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:19:56.463113 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.463082    2580 generic.go:356] "Generic (PLEG): container finished" podID="c502fa75-66c6-45a3-ad9f-7d0b28b3910f" containerID="4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f" exitCode=0
Sep 10 21:19:56.463317 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.463148    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"
Sep 10 21:19:56.463317 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.463166    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx" event={"ID":"c502fa75-66c6-45a3-ad9f-7d0b28b3910f","Type":"ContainerDied","Data":"4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f"}
Sep 10 21:19:56.463317 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.463219    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx" event={"ID":"c502fa75-66c6-45a3-ad9f-7d0b28b3910f","Type":"ContainerDied","Data":"75d4c2bb9671ec0d7baa2cc23c0c6271cb941037e12c7970f62c26fa60da855d"}
Sep 10 21:19:56.463317 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.463242    2580 scope.go:117] "RemoveContainer" containerID="4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f"
Sep 10 21:19:56.474048 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.474032    2580 scope.go:117] "RemoveContainer" containerID="4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f"
Sep 10 21:19:56.474395 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:19:56.474375    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f\": container with ID starting with 4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f not found: ID does not exist" containerID="4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f"
Sep 10 21:19:56.474499 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.474400    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f"} err="failed to get container status \"4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f\": rpc error: code = NotFound desc = could not find container \"4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f\": container with ID starting with 4de4a9b019397098ab93d9488415de6aa98ee282176eb698adead09c092fa43f not found: ID does not exist"
Sep 10 21:19:56.485414 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.485390    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"]
Sep 10 21:19:56.492486 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:56.492456    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7f376d02-655b68d6d6-khgsx"]
Sep 10 21:19:57.525417 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:19:57.525387    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c502fa75-66c6-45a3-ad9f-7d0b28b3910f" path="/var/lib/kubelet/pods/c502fa75-66c6-45a3-ad9f-7d0b28b3910f/volumes"
Sep 10 21:21:49.751130 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.751076    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"]
Sep 10 21:21:49.751720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.751567    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c502fa75-66c6-45a3-ad9f-7d0b28b3910f" containerName="payload-pre-processing"
Sep 10 21:21:49.751720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.751585    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c502fa75-66c6-45a3-ad9f-7d0b28b3910f" containerName="payload-pre-processing"
Sep 10 21:21:49.751720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.751651    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c502fa75-66c6-45a3-ad9f-7d0b28b3910f" containerName="payload-pre-processing"
Sep 10 21:21:49.753872 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.753852    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.758340 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.758320    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a249a2-serving-cert\""
Sep 10 21:21:49.767512 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.767489    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"]
Sep 10 21:21:49.888220 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.888189    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-metrics-certs\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.888407 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.888242    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-openshift-service-ca\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.888407 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.888348    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vrb\" (UniqueName: \"kubernetes.io/projected/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-kube-api-access-m7vrb\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.888407 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.888399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-maas-api-tls\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.989424 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.989387    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vrb\" (UniqueName: \"kubernetes.io/projected/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-kube-api-access-m7vrb\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.989600 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.989439    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-maas-api-tls\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.989600 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.989490    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-metrics-certs\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.989600 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.989529    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-openshift-service-ca\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.990302 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.990217    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-openshift-service-ca\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.992184 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.992161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-maas-api-tls\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.992275 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.992161    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-metrics-certs\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:49.998215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:49.998195    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vrb\" (UniqueName: \"kubernetes.io/projected/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-kube-api-access-m7vrb\") pod \"maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:50.066341 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:50.066273    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:50.346174 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:50.346143    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"]
Sep 10 21:21:50.348066 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:21:50.348036    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cdc2b7_af4f_4b08_8a1d_b9d949b32acf.slice/crio-d7aab9919b1f81b9b83656147ddfa20aaaee9c81bec11f90ce49d256b7d0448e WatchSource:0}: Error finding container d7aab9919b1f81b9b83656147ddfa20aaaee9c81bec11f90ce49d256b7d0448e: Status 404 returned error can't find the container with id d7aab9919b1f81b9b83656147ddfa20aaaee9c81bec11f90ce49d256b7d0448e
Sep 10 21:21:50.349549 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:50.349532    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:21:51.133514 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:51.133429    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w" event={"ID":"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf","Type":"ContainerStarted","Data":"5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445"}
Sep 10 21:21:51.133514 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:51.133486    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w" event={"ID":"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf","Type":"ContainerStarted","Data":"d7aab9919b1f81b9b83656147ddfa20aaaee9c81bec11f90ce49d256b7d0448e"}
Sep 10 21:21:51.133917 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:51.133646    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:21:51.154475 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:51.154429    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w" podStartSLOduration=1.62049674 podStartE2EDuration="2.154407174s" totalImagesPullingTime="533.910434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:21:51.152513745 +0000 UTC m=+1398.154760084" watchObservedRunningTime="2026-09-10 21:21:51.154407174 +0000 UTC m=+1398.156653520"
Sep 10 21:21:57.144713 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:21:57.144681    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:22:19.593309 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.593278    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"]
Sep 10 21:22:19.596987 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.596970    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:19.599888 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.599861    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cf7171\""
Sep 10 21:22:19.609228 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.609201    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"]
Sep 10 21:22:19.647140 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.647089    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"]
Sep 10 21:22:19.650542 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.650526    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:19.652614 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.652592    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cf7171-dockercfg-sm6vv\""
Sep 10 21:22:19.661707 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.661686    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"]
Sep 10 21:22:19.667730 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.667710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gvq\" (UniqueName: \"kubernetes.io/projected/79b9ea64-86a2-465e-adba-74ce179b52dc-kube-api-access-q8gvq\") pod \"payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn\" (UID: \"79b9ea64-86a2-465e-adba-74ce179b52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:19.667834 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.667819    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9ea64-86a2-465e-adba-74ce179b52dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn\" (UID: \"79b9ea64-86a2-465e-adba-74ce179b52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:19.768381 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.768350    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdwpl\" (UniqueName: \"kubernetes.io/projected/3376cb6f-fdf0-4813-9c86-d4f22591bf29-kube-api-access-wdwpl\") pod \"payload-processing-e2e-api-b-cf7171-777755584c-8qvhb\" (UID: \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:19.768616 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.768401    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9ea64-86a2-465e-adba-74ce179b52dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn\" (UID: \"79b9ea64-86a2-465e-adba-74ce179b52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:19.768616 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.768454    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gvq\" (UniqueName: \"kubernetes.io/projected/79b9ea64-86a2-465e-adba-74ce179b52dc-kube-api-access-q8gvq\") pod \"payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn\" (UID: \"79b9ea64-86a2-465e-adba-74ce179b52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:19.768616 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.768491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3376cb6f-fdf0-4813-9c86-d4f22591bf29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf7171-777755584c-8qvhb\" (UID: \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:19.769159 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.769113    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9ea64-86a2-465e-adba-74ce179b52dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn\" (UID: \"79b9ea64-86a2-465e-adba-74ce179b52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:19.777986 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.777957    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gvq\" (UniqueName: \"kubernetes.io/projected/79b9ea64-86a2-465e-adba-74ce179b52dc-kube-api-access-q8gvq\") pod \"payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn\" (UID: \"79b9ea64-86a2-465e-adba-74ce179b52dc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:19.869694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.869597    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3376cb6f-fdf0-4813-9c86-d4f22591bf29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf7171-777755584c-8qvhb\" (UID: \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:19.869694 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.869656    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdwpl\" (UniqueName: \"kubernetes.io/projected/3376cb6f-fdf0-4813-9c86-d4f22591bf29-kube-api-access-wdwpl\") pod \"payload-processing-e2e-api-b-cf7171-777755584c-8qvhb\" (UID: \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:19.870407 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.870382    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3376cb6f-fdf0-4813-9c86-d4f22591bf29-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cf7171-777755584c-8qvhb\" (UID: \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:19.878624 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.878600    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdwpl\" (UniqueName: \"kubernetes.io/projected/3376cb6f-fdf0-4813-9c86-d4f22591bf29-kube-api-access-wdwpl\") pod \"payload-processing-e2e-api-b-cf7171-777755584c-8qvhb\" (UID: \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\") " pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:19.908661 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.908636    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:19.961878 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:19.961842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:20.188077 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:20.188052    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"]
Sep 10 21:22:20.189672 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:22:20.189643    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b9ea64_86a2_465e_adba_74ce179b52dc.slice/crio-0643b38fe471134af0903f297b9f16865618464b2cda49813e54a5a47920806e WatchSource:0}: Error finding container 0643b38fe471134af0903f297b9f16865618464b2cda49813e54a5a47920806e: Status 404 returned error can't find the container with id 0643b38fe471134af0903f297b9f16865618464b2cda49813e54a5a47920806e
Sep 10 21:22:20.251372 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:20.251346    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"]
Sep 10 21:22:20.253459 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:22:20.253434    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3376cb6f_fdf0_4813_9c86_d4f22591bf29.slice/crio-117026b3718205627ad63815912c232e6ca067aebe8ed7fa8636cdfdb3d72185 WatchSource:0}: Error finding container 117026b3718205627ad63815912c232e6ca067aebe8ed7fa8636cdfdb3d72185: Status 404 returned error can't find the container with id 117026b3718205627ad63815912c232e6ca067aebe8ed7fa8636cdfdb3d72185
Sep 10 21:22:20.310582 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:20.310545    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn" event={"ID":"79b9ea64-86a2-465e-adba-74ce179b52dc","Type":"ContainerStarted","Data":"388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986"}
Sep 10 21:22:20.310734 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:20.310597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn" event={"ID":"79b9ea64-86a2-465e-adba-74ce179b52dc","Type":"ContainerStarted","Data":"0643b38fe471134af0903f297b9f16865618464b2cda49813e54a5a47920806e"}
Sep 10 21:22:20.310957 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:20.310936    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:20.312211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:20.312191    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb" event={"ID":"3376cb6f-fdf0-4813-9c86-d4f22591bf29","Type":"ContainerStarted","Data":"117026b3718205627ad63815912c232e6ca067aebe8ed7fa8636cdfdb3d72185"}
Sep 10 21:22:20.331855 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:20.331806    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn" podStartSLOduration=1.331791737 podStartE2EDuration="1.331791737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:20.330943987 +0000 UTC m=+1427.333190332" watchObservedRunningTime="2026-09-10 21:22:20.331791737 +0000 UTC m=+1427.334038106"
Sep 10 21:22:21.320288 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:21.320254    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb" event={"ID":"3376cb6f-fdf0-4813-9c86-d4f22591bf29","Type":"ContainerStarted","Data":"a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc"}
Sep 10 21:22:21.320767 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:21.320600    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:22:21.352190 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:21.352148    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb" podStartSLOduration=2.352111014 podStartE2EDuration="2.352111014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:22:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:22:21.350492635 +0000 UTC m=+1428.352738983" watchObservedRunningTime="2026-09-10 21:22:21.352111014 +0000 UTC m=+1428.354357360"
Sep 10 21:22:31.322202 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:31.322091    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:22:32.328809 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:22:32.328778    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:23:01.675056 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.675017    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"]
Sep 10 21:23:01.679619 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.675307    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w" podUID="c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" containerName="maas-api" containerID="cri-o://5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445" gracePeriod=30
Sep 10 21:23:01.932562 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.932498    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:23:01.970857 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.970828    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7vrb\" (UniqueName: \"kubernetes.io/projected/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-kube-api-access-m7vrb\") pod \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") "
Sep 10 21:23:01.971032 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.970909    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-openshift-service-ca\") pod \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") "
Sep 10 21:23:01.971032 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.970999    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-maas-api-tls\") pod \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") "
Sep 10 21:23:01.971180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.971104    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-metrics-certs\") pod \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\" (UID: \"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf\") "
Sep 10 21:23:01.971250 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.971226    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" (UID: "c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:01.971470 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.971454    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:01.973176 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.973152    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" (UID: "c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:23:01.973265 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.973189    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" (UID: "c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:23:01.973265 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:01.973250    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-kube-api-access-m7vrb" (OuterVolumeSpecName: "kube-api-access-m7vrb") pod "c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" (UID: "c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf"). InnerVolumeSpecName "kube-api-access-m7vrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:02.072339 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.072313    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:02.072339 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.072336    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:02.072339 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.072345    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7vrb\" (UniqueName: \"kubernetes.io/projected/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf-kube-api-access-m7vrb\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:02.574925 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.574894    2580 generic.go:356] "Generic (PLEG): container finished" podID="c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" containerID="5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445" exitCode=0
Sep 10 21:23:02.575200 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.574941    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"
Sep 10 21:23:02.575200 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.574972    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w" event={"ID":"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf","Type":"ContainerDied","Data":"5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445"}
Sep 10 21:23:02.575200 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.575030    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w" event={"ID":"c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf","Type":"ContainerDied","Data":"d7aab9919b1f81b9b83656147ddfa20aaaee9c81bec11f90ce49d256b7d0448e"}
Sep 10 21:23:02.575200 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.575069    2580 scope.go:117] "RemoveContainer" containerID="5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445"
Sep 10 21:23:02.585346 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.585330    2580 scope.go:117] "RemoveContainer" containerID="5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445"
Sep 10 21:23:02.585649 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:23:02.585625    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445\": container with ID starting with 5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445 not found: ID does not exist" containerID="5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445"
Sep 10 21:23:02.585711 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.585659    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445"} err="failed to get container status \"5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445\": rpc error: code = NotFound desc = could not find container \"5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445\": container with ID starting with 5e012a55b231809f8ce97b01a6ebad823223adec14c69f39c0d49e1685c59445 not found: ID does not exist"
Sep 10 21:23:02.599958 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.599933    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"]
Sep 10 21:23:02.607471 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:02.607445    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a249a2-f696d9bd5-blp4w"]
Sep 10 21:23:03.519975 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:03.519944    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" path="/var/lib/kubelet/pods/c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf/volumes"
Sep 10 21:23:25.111229 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.110889    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"]
Sep 10 21:23:25.111229 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.111175    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb" podUID="3376cb6f-fdf0-4813-9c86-d4f22591bf29" containerName="payload-processing" containerID="cri-o://a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc" gracePeriod=30
Sep 10 21:23:25.116577 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.116415    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"]
Sep 10 21:23:25.116718 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.116634    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn" podUID="79b9ea64-86a2-465e-adba-74ce179b52dc" containerName="payload-pre-processing" containerID="cri-o://388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986" gracePeriod=30
Sep 10 21:23:25.371377 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.371317    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:23:25.400949 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.400926    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:23:25.502197 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.502171    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9ea64-86a2-465e-adba-74ce179b52dc-plugins-config-volume\") pod \"79b9ea64-86a2-465e-adba-74ce179b52dc\" (UID: \"79b9ea64-86a2-465e-adba-74ce179b52dc\") "
Sep 10 21:23:25.502385 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.502211    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdwpl\" (UniqueName: \"kubernetes.io/projected/3376cb6f-fdf0-4813-9c86-d4f22591bf29-kube-api-access-wdwpl\") pod \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\" (UID: \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\") "
Sep 10 21:23:25.502385 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.502305    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3376cb6f-fdf0-4813-9c86-d4f22591bf29-plugins-config-volume\") pod \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\" (UID: \"3376cb6f-fdf0-4813-9c86-d4f22591bf29\") "
Sep 10 21:23:25.502385 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.502341    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8gvq\" (UniqueName: \"kubernetes.io/projected/79b9ea64-86a2-465e-adba-74ce179b52dc-kube-api-access-q8gvq\") pod \"79b9ea64-86a2-465e-adba-74ce179b52dc\" (UID: \"79b9ea64-86a2-465e-adba-74ce179b52dc\") "
Sep 10 21:23:25.502581 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.502555    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79b9ea64-86a2-465e-adba-74ce179b52dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "79b9ea64-86a2-465e-adba-74ce179b52dc" (UID: "79b9ea64-86a2-465e-adba-74ce179b52dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:25.502700 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.502677    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3376cb6f-fdf0-4813-9c86-d4f22591bf29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3376cb6f-fdf0-4813-9c86-d4f22591bf29" (UID: "3376cb6f-fdf0-4813-9c86-d4f22591bf29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:25.504564 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.504539    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79b9ea64-86a2-465e-adba-74ce179b52dc-kube-api-access-q8gvq" (OuterVolumeSpecName: "kube-api-access-q8gvq") pod "79b9ea64-86a2-465e-adba-74ce179b52dc" (UID: "79b9ea64-86a2-465e-adba-74ce179b52dc"). InnerVolumeSpecName "kube-api-access-q8gvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:25.504637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.504614    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3376cb6f-fdf0-4813-9c86-d4f22591bf29-kube-api-access-wdwpl" (OuterVolumeSpecName: "kube-api-access-wdwpl") pod "3376cb6f-fdf0-4813-9c86-d4f22591bf29" (UID: "3376cb6f-fdf0-4813-9c86-d4f22591bf29"). InnerVolumeSpecName "kube-api-access-wdwpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:25.603534 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.603501    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gvq\" (UniqueName: \"kubernetes.io/projected/79b9ea64-86a2-465e-adba-74ce179b52dc-kube-api-access-q8gvq\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:25.603534 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.603532    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79b9ea64-86a2-465e-adba-74ce179b52dc-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:25.603714 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.603546    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdwpl\" (UniqueName: \"kubernetes.io/projected/3376cb6f-fdf0-4813-9c86-d4f22591bf29-kube-api-access-wdwpl\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:25.603714 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.603560    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3376cb6f-fdf0-4813-9c86-d4f22591bf29-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:25.706627 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.706588    2580 generic.go:356] "Generic (PLEG): container finished" podID="79b9ea64-86a2-465e-adba-74ce179b52dc" containerID="388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986" exitCode=0
Sep 10 21:23:25.706811 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.706633    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"
Sep 10 21:23:25.706811 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.706640    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn" event={"ID":"79b9ea64-86a2-465e-adba-74ce179b52dc","Type":"ContainerDied","Data":"388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986"}
Sep 10 21:23:25.706811 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.706690    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn" event={"ID":"79b9ea64-86a2-465e-adba-74ce179b52dc","Type":"ContainerDied","Data":"0643b38fe471134af0903f297b9f16865618464b2cda49813e54a5a47920806e"}
Sep 10 21:23:25.706811 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.706712    2580 scope.go:117] "RemoveContainer" containerID="388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986"
Sep 10 21:23:25.708213 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.708190    2580 generic.go:356] "Generic (PLEG): container finished" podID="3376cb6f-fdf0-4813-9c86-d4f22591bf29" containerID="a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc" exitCode=0
Sep 10 21:23:25.708497 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.708233    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"
Sep 10 21:23:25.708497 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.708249    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb" event={"ID":"3376cb6f-fdf0-4813-9c86-d4f22591bf29","Type":"ContainerDied","Data":"a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc"}
Sep 10 21:23:25.708497 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.708288    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb" event={"ID":"3376cb6f-fdf0-4813-9c86-d4f22591bf29","Type":"ContainerDied","Data":"117026b3718205627ad63815912c232e6ca067aebe8ed7fa8636cdfdb3d72185"}
Sep 10 21:23:25.716009 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.715980    2580 scope.go:117] "RemoveContainer" containerID="388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986"
Sep 10 21:23:25.716303 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:23:25.716287    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986\": container with ID starting with 388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986 not found: ID does not exist" containerID="388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986"
Sep 10 21:23:25.716370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.716312    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986"} err="failed to get container status \"388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986\": rpc error: code = NotFound desc = could not find container \"388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986\": container with ID starting with 388eae4d87eaad37a92ceb2bf428393bb84d7323a52fb5bf4f0c3e67c9101986 not found: ID does not exist"
Sep 10 21:23:25.716370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.716352    2580 scope.go:117] "RemoveContainer" containerID="a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc"
Sep 10 21:23:25.725053 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.725007    2580 scope.go:117] "RemoveContainer" containerID="a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc"
Sep 10 21:23:25.725357 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:23:25.725339    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc\": container with ID starting with a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc not found: ID does not exist" containerID="a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc"
Sep 10 21:23:25.725431 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.725363    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc"} err="failed to get container status \"a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc\": rpc error: code = NotFound desc = could not find container \"a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc\": container with ID starting with a24fdd51902acb2b8a4ee0ff284d5523f4b568fcf0f60444e7571bbfcd741edc not found: ID does not exist"
Sep 10 21:23:25.730528 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.730430    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"]
Sep 10 21:23:25.736657 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.736630    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cf7171-585f9894f8-dd8rn"]
Sep 10 21:23:25.752569 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.752547    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"]
Sep 10 21:23:25.757517 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:25.757497    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cf7171-777755584c-8qvhb"]
Sep 10 21:23:27.519444 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:27.519413    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3376cb6f-fdf0-4813-9c86-d4f22591bf29" path="/var/lib/kubelet/pods/3376cb6f-fdf0-4813-9c86-d4f22591bf29/volumes"
Sep 10 21:23:27.519813 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:27.519750    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79b9ea64-86a2-465e-adba-74ce179b52dc" path="/var/lib/kubelet/pods/79b9ea64-86a2-465e-adba-74ce179b52dc/volumes"
Sep 10 21:23:43.437855 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.437823    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"]
Sep 10 21:23:43.438397 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.438062    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj" podUID="5d360019-bff8-47e6-994f-ab1b1800dc64" containerName="payload-pre-processing" containerID="cri-o://d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105" gracePeriod=30
Sep 10 21:23:43.693074 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.693004    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:23:43.772787 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.772761    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d360019-bff8-47e6-994f-ab1b1800dc64-plugins-config-volume\") pod \"5d360019-bff8-47e6-994f-ab1b1800dc64\" (UID: \"5d360019-bff8-47e6-994f-ab1b1800dc64\") "
Sep 10 21:23:43.772944 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.772836    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgp6t\" (UniqueName: \"kubernetes.io/projected/5d360019-bff8-47e6-994f-ab1b1800dc64-kube-api-access-bgp6t\") pod \"5d360019-bff8-47e6-994f-ab1b1800dc64\" (UID: \"5d360019-bff8-47e6-994f-ab1b1800dc64\") "
Sep 10 21:23:43.773103 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.773083    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d360019-bff8-47e6-994f-ab1b1800dc64-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d360019-bff8-47e6-994f-ab1b1800dc64" (UID: "5d360019-bff8-47e6-994f-ab1b1800dc64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:43.774986 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.774961    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d360019-bff8-47e6-994f-ab1b1800dc64-kube-api-access-bgp6t" (OuterVolumeSpecName: "kube-api-access-bgp6t") pod "5d360019-bff8-47e6-994f-ab1b1800dc64" (UID: "5d360019-bff8-47e6-994f-ab1b1800dc64"). InnerVolumeSpecName "kube-api-access-bgp6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:43.811380 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.811353    2580 generic.go:356] "Generic (PLEG): container finished" podID="5d360019-bff8-47e6-994f-ab1b1800dc64" containerID="d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105" exitCode=0
Sep 10 21:23:43.811504 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.811393    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"
Sep 10 21:23:43.811504 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.811431    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj" event={"ID":"5d360019-bff8-47e6-994f-ab1b1800dc64","Type":"ContainerDied","Data":"d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105"}
Sep 10 21:23:43.811504 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.811481    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj" event={"ID":"5d360019-bff8-47e6-994f-ab1b1800dc64","Type":"ContainerDied","Data":"3d841edc61f7f7bea1bbd1dbbbb86cb33f11e5d9a70071f310a00b391ded4e5b"}
Sep 10 21:23:43.811504 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.811503    2580 scope.go:117] "RemoveContainer" containerID="d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105"
Sep 10 21:23:43.821158 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.821143    2580 scope.go:117] "RemoveContainer" containerID="d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105"
Sep 10 21:23:43.821462 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:23:43.821444    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105\": container with ID starting with d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105 not found: ID does not exist" containerID="d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105"
Sep 10 21:23:43.821540 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.821471    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105"} err="failed to get container status \"d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105\": rpc error: code = NotFound desc = could not find container \"d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105\": container with ID starting with d1d8f864b78f175cc220c6644022c87deb0e0d4590111367e0e0d5375ad34105 not found: ID does not exist"
Sep 10 21:23:43.834949 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.834926    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"]
Sep 10 21:23:43.841337 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.841315    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-501c57-748548456b-zk5pj"]
Sep 10 21:23:43.873695 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.873674    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d360019-bff8-47e6-994f-ab1b1800dc64-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:43.873788 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:43.873697    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgp6t\" (UniqueName: \"kubernetes.io/projected/5d360019-bff8-47e6-994f-ab1b1800dc64-kube-api-access-bgp6t\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:45.519546 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:45.519512    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d360019-bff8-47e6-994f-ab1b1800dc64" path="/var/lib/kubelet/pods/5d360019-bff8-47e6-994f-ab1b1800dc64/volumes"
Sep 10 21:23:50.116558 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:50.116526    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"]
Sep 10 21:23:50.116951 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:50.116743    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy" containerID="cri-o://323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7" gracePeriod=30
Sep 10 21:23:55.376293 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.376254    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:23:55.486513 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486420    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-certs\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.486513 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486478    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-credential-socket\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.486513 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486509    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzb8z\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-kube-api-access-dzb8z\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.486853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486540    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-envoy\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.486853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486584    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed59769b-454d-4358-b0b2-2334cd4542a4-istiod-ca-cert\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.486853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486708    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-token\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.486853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486748    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-podinfo\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.486853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486752    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:55.486853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486820    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-socket\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.486853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486853    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-data\") pod \"ed59769b-454d-4358-b0b2-2334cd4542a4\" (UID: \"ed59769b-454d-4358-b0b2-2334cd4542a4\") "
Sep 10 21:23:55.487260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486816    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:55.487260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.486957    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed59769b-454d-4358-b0b2-2334cd4542a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:23:55.487260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.487072    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:55.487260 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.487248    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:55.487637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.487422    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.487637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.487443    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed59769b-454d-4358-b0b2-2334cd4542a4-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.487637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.487457    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.487637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.487470    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.489059 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.489027    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-kube-api-access-dzb8z" (OuterVolumeSpecName: "kube-api-access-dzb8z") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "kube-api-access-dzb8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:55.489059 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.489054    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:23:55.489243 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.489173    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:23:55.489313 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.489298    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed59769b-454d-4358-b0b2-2334cd4542a4" (UID: "ed59769b-454d-4358-b0b2-2334cd4542a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:23:55.588211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.588181    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.588211 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.588212    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.588464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.588226    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.588464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.588242    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzb8z\" (UniqueName: \"kubernetes.io/projected/ed59769b-454d-4358-b0b2-2334cd4542a4-kube-api-access-dzb8z\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.588464 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.588258    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed59769b-454d-4358-b0b2-2334cd4542a4-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:23:55.887393 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.887360    2580 generic.go:356] "Generic (PLEG): container finished" podID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerID="323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7" exitCode=0
Sep 10 21:23:55.887563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.887421    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"
Sep 10 21:23:55.887563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.887450    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" event={"ID":"ed59769b-454d-4358-b0b2-2334cd4542a4","Type":"ContainerDied","Data":"323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7"}
Sep 10 21:23:55.887563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.887508    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc" event={"ID":"ed59769b-454d-4358-b0b2-2334cd4542a4","Type":"ContainerDied","Data":"f17992b643ce23f129df9b5f503a9f35f3f2421c9dcf235158bc8b0fdc13dbd3"}
Sep 10 21:23:55.887563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.887532    2580 scope.go:117] "RemoveContainer" containerID="323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7"
Sep 10 21:23:55.896884 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.896869    2580 scope.go:117] "RemoveContainer" containerID="0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7"
Sep 10 21:23:55.905722 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.905705    2580 scope.go:117] "RemoveContainer" containerID="323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7"
Sep 10 21:23:55.906023 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:23:55.906002    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7\": container with ID starting with 323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7 not found: ID does not exist" containerID="323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7"
Sep 10 21:23:55.906133 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.906033    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7"} err="failed to get container status \"323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7\": rpc error: code = NotFound desc = could not find container \"323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7\": container with ID starting with 323a5545832e86487830c117c3a30738a6ceac18b623031b51b9939c1a6f40d7 not found: ID does not exist"
Sep 10 21:23:55.906133 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.906053    2580 scope.go:117] "RemoveContainer" containerID="0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7"
Sep 10 21:23:55.906537 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:23:55.906363    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7\": container with ID starting with 0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7 not found: ID does not exist" containerID="0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7"
Sep 10 21:23:55.906537 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.906395    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7"} err="failed to get container status \"0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7\": rpc error: code = NotFound desc = could not find container \"0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7\": container with ID starting with 0752ac40c34aa5737af3e0e68e825732e694b4166a9aec126d54eab0e50979e7 not found: ID does not exist"
Sep 10 21:23:55.906665 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.906646    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"]
Sep 10 21:23:55.914300 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:55.914270    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-501c57-openshift-default-5f6dc669bd-f45cc"]
Sep 10 21:23:57.519231 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:23:57.519195    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" path="/var/lib/kubelet/pods/ed59769b-454d-4358-b0b2-2334cd4542a4/volumes"
Sep 10 21:24:00.990106 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:00.990063    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"]
Sep 10 21:24:00.990589 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:00.990335    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9" podUID="10ad7576-2d43-4205-91dc-a98639086062" containerName="maas-api" containerID="cri-o://935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e" gracePeriod=30
Sep 10 21:24:01.042564 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.042533    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"]
Sep 10 21:24:01.042961 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.042780    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4" podUID="9f033e79-94fa-4e8b-ae82-1fb32f893121" containerName="payload-processing" containerID="cri-o://fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5" gracePeriod=30
Sep 10 21:24:01.300941 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.300916    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:24:01.341734 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.341703    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:24:01.436979 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.436948    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"]
Sep 10 21:24:01.437370 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437355    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d360019-bff8-47e6-994f-ab1b1800dc64" containerName="payload-pre-processing"
Sep 10 21:24:01.437421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437373    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d360019-bff8-47e6-994f-ab1b1800dc64" containerName="payload-pre-processing"
Sep 10 21:24:01.437421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437383    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" containerName="maas-api"
Sep 10 21:24:01.437421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437389    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" containerName="maas-api"
Sep 10 21:24:01.437421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437404    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3376cb6f-fdf0-4813-9c86-d4f22591bf29" containerName="payload-processing"
Sep 10 21:24:01.437421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437410    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3376cb6f-fdf0-4813-9c86-d4f22591bf29" containerName="payload-processing"
Sep 10 21:24:01.437421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437417    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy"
Sep 10 21:24:01.437421 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437422    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437428    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f033e79-94fa-4e8b-ae82-1fb32f893121" containerName="payload-processing"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437433    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f033e79-94fa-4e8b-ae82-1fb32f893121" containerName="payload-processing"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437442    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437448    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437454    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79b9ea64-86a2-465e-adba-74ce179b52dc" containerName="payload-pre-processing"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437459    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="79b9ea64-86a2-465e-adba-74ce179b52dc" containerName="payload-pre-processing"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437465    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10ad7576-2d43-4205-91dc-a98639086062" containerName="maas-api"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437471    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="10ad7576-2d43-4205-91dc-a98639086062" containerName="maas-api"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437522    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f033e79-94fa-4e8b-ae82-1fb32f893121" containerName="payload-processing"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437530    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1cdc2b7-af4f-4b08-8a1d-b9d949b32acf" containerName="maas-api"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437535    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="10ad7576-2d43-4205-91dc-a98639086062" containerName="maas-api"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437543    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="79b9ea64-86a2-465e-adba-74ce179b52dc" containerName="payload-pre-processing"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437548    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3376cb6f-fdf0-4813-9c86-d4f22591bf29" containerName="payload-processing"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437555    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437559    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed59769b-454d-4358-b0b2-2334cd4542a4" containerName="istio-proxy"
Sep 10 21:24:01.437678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.437564    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d360019-bff8-47e6-994f-ab1b1800dc64" containerName="payload-pre-processing"
Sep 10 21:24:01.440338 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.440316    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-957r5\" (UniqueName: \"kubernetes.io/projected/9f033e79-94fa-4e8b-ae82-1fb32f893121-kube-api-access-957r5\") pod \"9f033e79-94fa-4e8b-ae82-1fb32f893121\" (UID: \"9f033e79-94fa-4e8b-ae82-1fb32f893121\") "
Sep 10 21:24:01.440466 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.440367    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f033e79-94fa-4e8b-ae82-1fb32f893121-plugins-config-volume\") pod \"9f033e79-94fa-4e8b-ae82-1fb32f893121\" (UID: \"9f033e79-94fa-4e8b-ae82-1fb32f893121\") "
Sep 10 21:24:01.440530 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.440475    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls\") pod \"10ad7576-2d43-4205-91dc-a98639086062\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") "
Sep 10 21:24:01.440613 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.440596    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4ckc\" (UniqueName: \"kubernetes.io/projected/10ad7576-2d43-4205-91dc-a98639086062-kube-api-access-r4ckc\") pod \"10ad7576-2d43-4205-91dc-a98639086062\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") "
Sep 10 21:24:01.440680 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.440624    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-metrics-certs\") pod \"10ad7576-2d43-4205-91dc-a98639086062\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") "
Sep 10 21:24:01.440680 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.440647    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ad7576-2d43-4205-91dc-a98639086062-openshift-service-ca\") pod \"10ad7576-2d43-4205-91dc-a98639086062\" (UID: \"10ad7576-2d43-4205-91dc-a98639086062\") "
Sep 10 21:24:01.440784 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.440704    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.440784 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.440735    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f033e79-94fa-4e8b-ae82-1fb32f893121-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f033e79-94fa-4e8b-ae82-1fb32f893121" (UID: "9f033e79-94fa-4e8b-ae82-1fb32f893121"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:24:01.441036 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.441011    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ad7576-2d43-4205-91dc-a98639086062-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "10ad7576-2d43-4205-91dc-a98639086062" (UID: "10ad7576-2d43-4205-91dc-a98639086062"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:24:01.441166 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.441018    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f033e79-94fa-4e8b-ae82-1fb32f893121-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:24:01.442843 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.442819    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "10ad7576-2d43-4205-91dc-a98639086062" (UID: "10ad7576-2d43-4205-91dc-a98639086062"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:01.443032 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.443008    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3bbf7d-serving-cert\""
Sep 10 21:24:01.443163 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.443078    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f033e79-94fa-4e8b-ae82-1fb32f893121-kube-api-access-957r5" (OuterVolumeSpecName: "kube-api-access-957r5") pod "9f033e79-94fa-4e8b-ae82-1fb32f893121" (UID: "9f033e79-94fa-4e8b-ae82-1fb32f893121"). InnerVolumeSpecName "kube-api-access-957r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:01.443587 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.443554    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "10ad7576-2d43-4205-91dc-a98639086062" (UID: "10ad7576-2d43-4205-91dc-a98639086062"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:24:01.443587 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.443566    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ad7576-2d43-4205-91dc-a98639086062-kube-api-access-r4ckc" (OuterVolumeSpecName: "kube-api-access-r4ckc") pod "10ad7576-2d43-4205-91dc-a98639086062" (UID: "10ad7576-2d43-4205-91dc-a98639086062"). InnerVolumeSpecName "kube-api-access-r4ckc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:24:01.452175 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.452156    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"]
Sep 10 21:24:01.542528 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542492    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-467h4\" (UniqueName: \"kubernetes.io/projected/fe2b173d-141f-43b9-a43d-e724f9dd6c97-kube-api-access-467h4\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.542528 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542531    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.542695 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542674    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2b173d-141f-43b9-a43d-e724f9dd6c97-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.542743 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542710    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-metrics-certs\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.542797 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542785    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:24:01.542839 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542804    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4ckc\" (UniqueName: \"kubernetes.io/projected/10ad7576-2d43-4205-91dc-a98639086062-kube-api-access-r4ckc\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:24:01.542839 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542818    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/10ad7576-2d43-4205-91dc-a98639086062-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:24:01.542839 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542826    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ad7576-2d43-4205-91dc-a98639086062-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:24:01.542839 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.542834    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957r5\" (UniqueName: \"kubernetes.io/projected/9f033e79-94fa-4e8b-ae82-1fb32f893121-kube-api-access-957r5\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:24:01.643735 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.643700    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-467h4\" (UniqueName: \"kubernetes.io/projected/fe2b173d-141f-43b9-a43d-e724f9dd6c97-kube-api-access-467h4\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.643735 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.643734    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.643989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.643818    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2b173d-141f-43b9-a43d-e724f9dd6c97-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.643989 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.643849    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-metrics-certs\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.643989 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:24:01.643934    2580 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-serving-cert: secret "maas-api-e2e-ipp-a-3bbf7d-serving-cert" not found
Sep 10 21:24:01.644187 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:24:01.644040    2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls podName:fe2b173d-141f-43b9-a43d-e724f9dd6c97 nodeName:}" failed. No retries permitted until 2026-09-10 21:24:02.144017192 +0000 UTC m=+1529.146263519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls") pod "maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8" (UID: "fe2b173d-141f-43b9-a43d-e724f9dd6c97") : secret "maas-api-e2e-ipp-a-3bbf7d-serving-cert" not found
Sep 10 21:24:01.644497 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.644479    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2b173d-141f-43b9-a43d-e724f9dd6c97-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.646419 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.646401    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-metrics-certs\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.651457 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.651433    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-467h4\" (UniqueName: \"kubernetes.io/projected/fe2b173d-141f-43b9-a43d-e724f9dd6c97-kube-api-access-467h4\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:01.930194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.930154    2580 generic.go:356] "Generic (PLEG): container finished" podID="9f033e79-94fa-4e8b-ae82-1fb32f893121" containerID="fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5" exitCode=0
Sep 10 21:24:01.930412 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.930251    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4" event={"ID":"9f033e79-94fa-4e8b-ae82-1fb32f893121","Type":"ContainerDied","Data":"fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5"}
Sep 10 21:24:01.930412 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.930264    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"
Sep 10 21:24:01.930412 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.930337    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4" event={"ID":"9f033e79-94fa-4e8b-ae82-1fb32f893121","Type":"ContainerDied","Data":"e9972bf63bed6badb9cb8afe3f765fb74dfed932b0a9b59721ace74de7380b5e"}
Sep 10 21:24:01.930412 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.930367    2580 scope.go:117] "RemoveContainer" containerID="fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5"
Sep 10 21:24:01.931822 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.931759    2580 generic.go:356] "Generic (PLEG): container finished" podID="10ad7576-2d43-4205-91dc-a98639086062" containerID="935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e" exitCode=0
Sep 10 21:24:01.931822 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.931800    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9" event={"ID":"10ad7576-2d43-4205-91dc-a98639086062","Type":"ContainerDied","Data":"935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e"}
Sep 10 21:24:01.931952 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.931858    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9" event={"ID":"10ad7576-2d43-4205-91dc-a98639086062","Type":"ContainerDied","Data":"57234be7e7891dfb329197e4942b20590f82b3de162e674a78bfe970b3c61195"}
Sep 10 21:24:01.931952 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.931811    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"
Sep 10 21:24:01.940040 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.939999    2580 scope.go:117] "RemoveContainer" containerID="fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5"
Sep 10 21:24:01.940346 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:24:01.940323    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5\": container with ID starting with fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5 not found: ID does not exist" containerID="fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5"
Sep 10 21:24:01.940438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.940351    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5"} err="failed to get container status \"fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5\": rpc error: code = NotFound desc = could not find container \"fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5\": container with ID starting with fd9c12e8c0cd5c00358189a107b777d92b6632243655c5610aecfb8dc43f0df5 not found: ID does not exist"
Sep 10 21:24:01.940438 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.940371    2580 scope.go:117] "RemoveContainer" containerID="935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e"
Sep 10 21:24:01.948913 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.948897    2580 scope.go:117] "RemoveContainer" containerID="935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e"
Sep 10 21:24:01.949200 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:24:01.949177    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e\": container with ID starting with 935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e not found: ID does not exist" containerID="935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e"
Sep 10 21:24:01.949289 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.949209    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e"} err="failed to get container status \"935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e\": rpc error: code = NotFound desc = could not find container \"935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e\": container with ID starting with 935e6874297847989a4fde8310daedac4fbbbc07a5a3ed83c9f8cf2384e5da0e not found: ID does not exist"
Sep 10 21:24:01.953623 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.953599    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"]
Sep 10 21:24:01.966342 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.966322    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-4ad710-7cb47f69cc-fckl4"]
Sep 10 21:24:01.978848 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.978815    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"]
Sep 10 21:24:01.983157 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:01.983111    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-4ad710-55574c9d75-6stw9"]
Sep 10 21:24:02.148897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.148866    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:02.150081 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.150055    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"]
Sep 10 21:24:02.151400 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.151376    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:02.154938 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.154920    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.157215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.157199    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3bbf7d-dockercfg-9vxnn\""
Sep 10 21:24:02.157215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.157204    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3bbf7d\""
Sep 10 21:24:02.163205 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.163182    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"]
Sep 10 21:24:02.250220 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.250109    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc075b7-f237-4e46-99f9-cfdee083af49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25\" (UID: \"1dc075b7-f237-4e46-99f9-cfdee083af49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.250384 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.250223    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt7cg\" (UniqueName: \"kubernetes.io/projected/1dc075b7-f237-4e46-99f9-cfdee083af49-kube-api-access-pt7cg\") pod \"payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25\" (UID: \"1dc075b7-f237-4e46-99f9-cfdee083af49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.350968 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.350931    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc075b7-f237-4e46-99f9-cfdee083af49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25\" (UID: \"1dc075b7-f237-4e46-99f9-cfdee083af49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.350968 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.350969    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt7cg\" (UniqueName: \"kubernetes.io/projected/1dc075b7-f237-4e46-99f9-cfdee083af49-kube-api-access-pt7cg\") pod \"payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25\" (UID: \"1dc075b7-f237-4e46-99f9-cfdee083af49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.351642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.351623    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc075b7-f237-4e46-99f9-cfdee083af49-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25\" (UID: \"1dc075b7-f237-4e46-99f9-cfdee083af49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.353873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.353842    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:02.358798 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.358777    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt7cg\" (UniqueName: \"kubernetes.io/projected/1dc075b7-f237-4e46-99f9-cfdee083af49-kube-api-access-pt7cg\") pod \"payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25\" (UID: \"1dc075b7-f237-4e46-99f9-cfdee083af49\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.469675 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.469645    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.628556 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.628531    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"]
Sep 10 21:24:02.629917 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:24:02.629890    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2b173d_141f_43b9_a43d_e724f9dd6c97.slice/crio-af187aa249c36fc26abfdd5b3f8dfee9e187511587727d8e4bede4b9a7362019 WatchSource:0}: Error finding container af187aa249c36fc26abfdd5b3f8dfee9e187511587727d8e4bede4b9a7362019: Status 404 returned error can't find the container with id af187aa249c36fc26abfdd5b3f8dfee9e187511587727d8e4bede4b9a7362019
Sep 10 21:24:02.747809 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.747779    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"]
Sep 10 21:24:02.749689 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:24:02.749660    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc075b7_f237_4e46_99f9_cfdee083af49.slice/crio-fa2e8f5d72d89714773c45351f198659645152f3b7b4c794614d6f404df490ba WatchSource:0}: Error finding container fa2e8f5d72d89714773c45351f198659645152f3b7b4c794614d6f404df490ba: Status 404 returned error can't find the container with id fa2e8f5d72d89714773c45351f198659645152f3b7b4c794614d6f404df490ba
Sep 10 21:24:02.938936 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.938902    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8" event={"ID":"fe2b173d-141f-43b9-a43d-e724f9dd6c97","Type":"ContainerStarted","Data":"af187aa249c36fc26abfdd5b3f8dfee9e187511587727d8e4bede4b9a7362019"}
Sep 10 21:24:02.940687 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.940645    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25" event={"ID":"1dc075b7-f237-4e46-99f9-cfdee083af49","Type":"ContainerStarted","Data":"f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455"}
Sep 10 21:24:02.940817 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.940700    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25" event={"ID":"1dc075b7-f237-4e46-99f9-cfdee083af49","Type":"ContainerStarted","Data":"fa2e8f5d72d89714773c45351f198659645152f3b7b4c794614d6f404df490ba"}
Sep 10 21:24:02.940817 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.940766    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:02.959904 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:02.959850    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25" podStartSLOduration=0.959829666 podStartE2EDuration="959.829666ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:02.957331693 +0000 UTC m=+1529.959578041" watchObservedRunningTime="2026-09-10 21:24:02.959829666 +0000 UTC m=+1529.962076020"
Sep 10 21:24:03.520931 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:03.520895    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10ad7576-2d43-4205-91dc-a98639086062" path="/var/lib/kubelet/pods/10ad7576-2d43-4205-91dc-a98639086062/volumes"
Sep 10 21:24:03.521299 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:03.521283    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f033e79-94fa-4e8b-ae82-1fb32f893121" path="/var/lib/kubelet/pods/9f033e79-94fa-4e8b-ae82-1fb32f893121/volumes"
Sep 10 21:24:03.949801 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:03.949765    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8" event={"ID":"fe2b173d-141f-43b9-a43d-e724f9dd6c97","Type":"ContainerStarted","Data":"d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf"}
Sep 10 21:24:03.950020 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:03.949997    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:03.969000 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:03.968946    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8" podStartSLOduration=2.49922008 podStartE2EDuration="2.968932898s" totalImagesPullingTime="469.712818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:03.967400839 +0000 UTC m=+1530.969647186" watchObservedRunningTime="2026-09-10 21:24:03.968932898 +0000 UTC m=+1530.971179244"
Sep 10 21:24:09.960467 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:09.960432    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:24:13.952307 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:13.952275    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:24:31.206057 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.206019    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"]
Sep 10 21:24:31.210913 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.210881    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:31.213434 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.213412    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a581b5-dockercfg-l9ks6\""
Sep 10 21:24:31.213520 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.213438    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a581b5\""
Sep 10 21:24:31.224683 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.224650    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"]
Sep 10 21:24:31.306776 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.306733    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40ec1b8-e8ee-4aff-916c-b47891684186-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk\" (UID: \"f40ec1b8-e8ee-4aff-916c-b47891684186\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:31.306943 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.306782    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb289\" (UniqueName: \"kubernetes.io/projected/f40ec1b8-e8ee-4aff-916c-b47891684186-kube-api-access-lb289\") pod \"payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk\" (UID: \"f40ec1b8-e8ee-4aff-916c-b47891684186\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:31.407305 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.407266    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40ec1b8-e8ee-4aff-916c-b47891684186-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk\" (UID: \"f40ec1b8-e8ee-4aff-916c-b47891684186\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:31.407305 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.407308    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb289\" (UniqueName: \"kubernetes.io/projected/f40ec1b8-e8ee-4aff-916c-b47891684186-kube-api-access-lb289\") pod \"payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk\" (UID: \"f40ec1b8-e8ee-4aff-916c-b47891684186\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:31.408215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.408188    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40ec1b8-e8ee-4aff-916c-b47891684186-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk\" (UID: \"f40ec1b8-e8ee-4aff-916c-b47891684186\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:31.415785 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.415763    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb289\" (UniqueName: \"kubernetes.io/projected/f40ec1b8-e8ee-4aff-916c-b47891684186-kube-api-access-lb289\") pod \"payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk\" (UID: \"f40ec1b8-e8ee-4aff-916c-b47891684186\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:31.523637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.523557    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:31.798237 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:31.798211    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"]
Sep 10 21:24:31.799730 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:24:31.799707    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40ec1b8_e8ee_4aff_916c_b47891684186.slice/crio-49320a9f4c68af0a25e97c7834ba3677154d0950479fb2089374b1df6f19598e WatchSource:0}: Error finding container 49320a9f4c68af0a25e97c7834ba3677154d0950479fb2089374b1df6f19598e: Status 404 returned error can't find the container with id 49320a9f4c68af0a25e97c7834ba3677154d0950479fb2089374b1df6f19598e
Sep 10 21:24:32.120349 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:32.120263    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk" event={"ID":"f40ec1b8-e8ee-4aff-916c-b47891684186","Type":"ContainerStarted","Data":"725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b"}
Sep 10 21:24:32.120519 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:32.120431    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:32.120519 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:32.120487    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk" event={"ID":"f40ec1b8-e8ee-4aff-916c-b47891684186","Type":"ContainerStarted","Data":"49320a9f4c68af0a25e97c7834ba3677154d0950479fb2089374b1df6f19598e"}
Sep 10 21:24:32.139202 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:32.139154    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk" podStartSLOduration=1.1391142 podStartE2EDuration="1.1391142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:24:32.138286136 +0000 UTC m=+1559.140532484" watchObservedRunningTime="2026-09-10 21:24:32.1391142 +0000 UTC m=+1559.141360547"
Sep 10 21:24:43.129105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:43.129077    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:24:59.709919 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:59.709892    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mg8df_1aa96712-b017-464c-b322-0ef8cbd99c8f/payload-processing/0.log"
Sep 10 21:24:59.833753 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:24:59.833729    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk_f40ec1b8-e8ee-4aff-916c-b47891684186/payload-processing/0.log"
Sep 10 21:25:35.545325 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:35.545238    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25_1dc075b7-f237-4e46-99f9-cfdee083af49/payload-processing/0.log"
Sep 10 21:25:35.670833 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:35.670798    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mg8df_1aa96712-b017-464c-b322-0ef8cbd99c8f/payload-processing/0.log"
Sep 10 21:25:35.792648 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:35.792624    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk_f40ec1b8-e8ee-4aff-916c-b47891684186/payload-processing/0.log"
Sep 10 21:25:41.041974 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.041943    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"]
Sep 10 21:25:41.045489 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.045465    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.047837 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.047819    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-d24c8a-serving-cert\""
Sep 10 21:25:41.057016 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.056992    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"]
Sep 10 21:25:41.124707 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.124676    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.124873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.124713    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.124873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.124753    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkxjs\" (UniqueName: \"kubernetes.io/projected/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-kube-api-access-wkxjs\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.124873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.124778    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.225476 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.225436    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkxjs\" (UniqueName: \"kubernetes.io/projected/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-kube-api-access-wkxjs\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.225476 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.225476    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.225708 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.225559    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.225708 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.225586    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.226374 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.226309    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.228059 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.228040    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.228227 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.228206    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.233853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.233825    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkxjs\" (UniqueName: \"kubernetes.io/projected/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-kube-api-access-wkxjs\") pod \"maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.359324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.359241    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:41.636641 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:41.636614    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"]
Sep 10 21:25:41.637796 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:25:41.637770    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod898ca11d_13b8_4ff3_be94_ee9cd72eaa9e.slice/crio-68e10cdcb44835ff642b4f4f86f5c4e0c20b17d2d92446c7ccfd59539c3a7cd4 WatchSource:0}: Error finding container 68e10cdcb44835ff642b4f4f86f5c4e0c20b17d2d92446c7ccfd59539c3a7cd4: Status 404 returned error can't find the container with id 68e10cdcb44835ff642b4f4f86f5c4e0c20b17d2d92446c7ccfd59539c3a7cd4
Sep 10 21:25:42.512019 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:42.511982    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt" event={"ID":"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e","Type":"ContainerStarted","Data":"eccf97243de7809bdc4fabbf3527cb0dc8d5dad7d0abe802b4edd1ab535d2ab9"}
Sep 10 21:25:42.512443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:42.512043    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt" event={"ID":"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e","Type":"ContainerStarted","Data":"68e10cdcb44835ff642b4f4f86f5c4e0c20b17d2d92446c7ccfd59539c3a7cd4"}
Sep 10 21:25:42.512443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:42.512072    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:25:42.530691 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:42.530645    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt" podStartSLOduration=1.192310954 podStartE2EDuration="1.53062631s" totalImagesPullingTime="338.315356ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:25:42.529586658 +0000 UTC m=+1629.531833006" watchObservedRunningTime="2026-09-10 21:25:42.53062631 +0000 UTC m=+1629.532872659"
Sep 10 21:25:48.524108 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:25:48.524081    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:26:11.473144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.473097    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"]
Sep 10 21:26:11.473639 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.473337    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt" podUID="898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" containerName="maas-api" containerID="cri-o://eccf97243de7809bdc4fabbf3527cb0dc8d5dad7d0abe802b4edd1ab535d2ab9" gracePeriod=30
Sep 10 21:26:11.678579 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.678544    2580 generic.go:356] "Generic (PLEG): container finished" podID="898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" containerID="eccf97243de7809bdc4fabbf3527cb0dc8d5dad7d0abe802b4edd1ab535d2ab9" exitCode=0
Sep 10 21:26:11.678760 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.678721    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt" event={"ID":"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e","Type":"ContainerDied","Data":"eccf97243de7809bdc4fabbf3527cb0dc8d5dad7d0abe802b4edd1ab535d2ab9"}
Sep 10 21:26:11.732768 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.732699    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:26:11.785891 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.785863    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkxjs\" (UniqueName: \"kubernetes.io/projected/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-kube-api-access-wkxjs\") pod \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") "
Sep 10 21:26:11.786061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.785944    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-maas-api-tls\") pod \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") "
Sep 10 21:26:11.786061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.785972    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-openshift-service-ca\") pod \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") "
Sep 10 21:26:11.786061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.786009    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-metrics-certs\") pod \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\" (UID: \"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e\") "
Sep 10 21:26:11.786413 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.786390    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" (UID: "898ca11d-13b8-4ff3-be94-ee9cd72eaa9e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:11.788303 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.788277    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" (UID: "898ca11d-13b8-4ff3-be94-ee9cd72eaa9e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:11.788398 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.788317    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" (UID: "898ca11d-13b8-4ff3-be94-ee9cd72eaa9e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:11.788459 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.788423    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-kube-api-access-wkxjs" (OuterVolumeSpecName: "kube-api-access-wkxjs") pod "898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" (UID: "898ca11d-13b8-4ff3-be94-ee9cd72eaa9e"). InnerVolumeSpecName "kube-api-access-wkxjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:11.887573 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.887545    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:11.887573 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.887572    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkxjs\" (UniqueName: \"kubernetes.io/projected/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-kube-api-access-wkxjs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:11.887778 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.887582    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:11.887778 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:11.887591    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:12.685728 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:12.685700    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"
Sep 10 21:26:12.686171 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:12.685735    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt" event={"ID":"898ca11d-13b8-4ff3-be94-ee9cd72eaa9e","Type":"ContainerDied","Data":"68e10cdcb44835ff642b4f4f86f5c4e0c20b17d2d92446c7ccfd59539c3a7cd4"}
Sep 10 21:26:12.686171 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:12.685793    2580 scope.go:117] "RemoveContainer" containerID="eccf97243de7809bdc4fabbf3527cb0dc8d5dad7d0abe802b4edd1ab535d2ab9"
Sep 10 21:26:12.712606 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:12.712582    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"]
Sep 10 21:26:12.716192 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:12.716164    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-d24c8a-558c858784-fzfxt"]
Sep 10 21:26:13.519527 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:13.519484    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" path="/var/lib/kubelet/pods/898ca11d-13b8-4ff3-be94-ee9cd72eaa9e/volumes"
Sep 10 21:26:37.938818 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:37.938782    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"]
Sep 10 21:26:37.939322 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:37.938986    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8" podUID="fe2b173d-141f-43b9-a43d-e724f9dd6c97" containerName="maas-api" containerID="cri-o://d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf" gracePeriod=30
Sep 10 21:26:38.026467 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.026427    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"]
Sep 10 21:26:38.026712 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.026684    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25" podUID="1dc075b7-f237-4e46-99f9-cfdee083af49" containerName="payload-processing" containerID="cri-o://f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455" gracePeriod=30
Sep 10 21:26:38.293384 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.293353    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:26:38.311969 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.311946    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:26:38.440992 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.440915    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pt7cg\" (UniqueName: \"kubernetes.io/projected/1dc075b7-f237-4e46-99f9-cfdee083af49-kube-api-access-pt7cg\") pod \"1dc075b7-f237-4e46-99f9-cfdee083af49\" (UID: \"1dc075b7-f237-4e46-99f9-cfdee083af49\") "
Sep 10 21:26:38.440992 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.440974    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-metrics-certs\") pod \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") "
Sep 10 21:26:38.441267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.440995    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc075b7-f237-4e46-99f9-cfdee083af49-plugins-config-volume\") pod \"1dc075b7-f237-4e46-99f9-cfdee083af49\" (UID: \"1dc075b7-f237-4e46-99f9-cfdee083af49\") "
Sep 10 21:26:38.441267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.441021    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2b173d-141f-43b9-a43d-e724f9dd6c97-openshift-service-ca\") pod \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") "
Sep 10 21:26:38.441267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.441039    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls\") pod \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") "
Sep 10 21:26:38.441267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.441062    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-467h4\" (UniqueName: \"kubernetes.io/projected/fe2b173d-141f-43b9-a43d-e724f9dd6c97-kube-api-access-467h4\") pod \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\" (UID: \"fe2b173d-141f-43b9-a43d-e724f9dd6c97\") "
Sep 10 21:26:38.441473 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.441417    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc075b7-f237-4e46-99f9-cfdee083af49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1dc075b7-f237-4e46-99f9-cfdee083af49" (UID: "1dc075b7-f237-4e46-99f9-cfdee083af49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:38.441473 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.441448    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2b173d-141f-43b9-a43d-e724f9dd6c97-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fe2b173d-141f-43b9-a43d-e724f9dd6c97" (UID: "fe2b173d-141f-43b9-a43d-e724f9dd6c97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:26:38.443406 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.443383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fe2b173d-141f-43b9-a43d-e724f9dd6c97" (UID: "fe2b173d-141f-43b9-a43d-e724f9dd6c97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:38.443482 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.443400    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2b173d-141f-43b9-a43d-e724f9dd6c97-kube-api-access-467h4" (OuterVolumeSpecName: "kube-api-access-467h4") pod "fe2b173d-141f-43b9-a43d-e724f9dd6c97" (UID: "fe2b173d-141f-43b9-a43d-e724f9dd6c97"). InnerVolumeSpecName "kube-api-access-467h4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:38.443678 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.443659    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fe2b173d-141f-43b9-a43d-e724f9dd6c97" (UID: "fe2b173d-141f-43b9-a43d-e724f9dd6c97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:26:38.443741 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.443670    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc075b7-f237-4e46-99f9-cfdee083af49-kube-api-access-pt7cg" (OuterVolumeSpecName: "kube-api-access-pt7cg") pod "1dc075b7-f237-4e46-99f9-cfdee083af49" (UID: "1dc075b7-f237-4e46-99f9-cfdee083af49"). InnerVolumeSpecName "kube-api-access-pt7cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:26:38.542433 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.542402    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe2b173d-141f-43b9-a43d-e724f9dd6c97-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:38.542433 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.542425    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:38.542597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.542443    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-467h4\" (UniqueName: \"kubernetes.io/projected/fe2b173d-141f-43b9-a43d-e724f9dd6c97-kube-api-access-467h4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:38.542597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.542453    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt7cg\" (UniqueName: \"kubernetes.io/projected/1dc075b7-f237-4e46-99f9-cfdee083af49-kube-api-access-pt7cg\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:38.542597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.542460    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe2b173d-141f-43b9-a43d-e724f9dd6c97-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:38.542597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.542469    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dc075b7-f237-4e46-99f9-cfdee083af49-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:26:38.831784 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.831696    2580 generic.go:356] "Generic (PLEG): container finished" podID="fe2b173d-141f-43b9-a43d-e724f9dd6c97" containerID="d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf" exitCode=0
Sep 10 21:26:38.831784 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.831739    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"
Sep 10 21:26:38.832009 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.831789    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8" event={"ID":"fe2b173d-141f-43b9-a43d-e724f9dd6c97","Type":"ContainerDied","Data":"d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf"}
Sep 10 21:26:38.832009 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.831846    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8" event={"ID":"fe2b173d-141f-43b9-a43d-e724f9dd6c97","Type":"ContainerDied","Data":"af187aa249c36fc26abfdd5b3f8dfee9e187511587727d8e4bede4b9a7362019"}
Sep 10 21:26:38.832009 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.831863    2580 scope.go:117] "RemoveContainer" containerID="d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf"
Sep 10 21:26:38.833108 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.833088    2580 generic.go:356] "Generic (PLEG): container finished" podID="1dc075b7-f237-4e46-99f9-cfdee083af49" containerID="f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455" exitCode=0
Sep 10 21:26:38.833245 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.833194    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"
Sep 10 21:26:38.833245 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.833200    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25" event={"ID":"1dc075b7-f237-4e46-99f9-cfdee083af49","Type":"ContainerDied","Data":"f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455"}
Sep 10 21:26:38.833245 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.833240    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25" event={"ID":"1dc075b7-f237-4e46-99f9-cfdee083af49","Type":"ContainerDied","Data":"fa2e8f5d72d89714773c45351f198659645152f3b7b4c794614d6f404df490ba"}
Sep 10 21:26:38.842906 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.842890    2580 scope.go:117] "RemoveContainer" containerID="d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf"
Sep 10 21:26:38.843206 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:26:38.843141    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf\": container with ID starting with d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf not found: ID does not exist" containerID="d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf"
Sep 10 21:26:38.843206 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.843167    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf"} err="failed to get container status \"d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf\": rpc error: code = NotFound desc = could not find container \"d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf\": container with ID starting with d9eee8f92ca2d158b34f3ed86ed27f7972699a447a5106a80a75400fe6ab23cf not found: ID does not exist"
Sep 10 21:26:38.843206 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.843186    2580 scope.go:117] "RemoveContainer" containerID="f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455"
Sep 10 21:26:38.851242 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.851227    2580 scope.go:117] "RemoveContainer" containerID="f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455"
Sep 10 21:26:38.851511 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:26:38.851495    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455\": container with ID starting with f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455 not found: ID does not exist" containerID="f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455"
Sep 10 21:26:38.851552 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.851517    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455"} err="failed to get container status \"f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455\": rpc error: code = NotFound desc = could not find container \"f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455\": container with ID starting with f20f164e89c6e0bc9ffb88b62e2b8c2b1d2bc5fa2442b2ffe794960941bcf455 not found: ID does not exist"
Sep 10 21:26:38.854954 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.854931    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"]
Sep 10 21:26:38.858644 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.858623    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3bbf7d-7cf584d58f-qgnt8"]
Sep 10 21:26:38.867962 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.867942    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"]
Sep 10 21:26:38.871792 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:38.871769    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-3bbf7d-546ffd6957-jnf25"]
Sep 10 21:26:39.518902 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:39.518873    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dc075b7-f237-4e46-99f9-cfdee083af49" path="/var/lib/kubelet/pods/1dc075b7-f237-4e46-99f9-cfdee083af49/volumes"
Sep 10 21:26:39.519275 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:26:39.519262    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe2b173d-141f-43b9-a43d-e724f9dd6c97" path="/var/lib/kubelet/pods/fe2b173d-141f-43b9-a43d-e724f9dd6c97/volumes"
Sep 10 21:27:44.421131 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.421080    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"]
Sep 10 21:27:44.421655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.421376    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk" podUID="f40ec1b8-e8ee-4aff-916c-b47891684186" containerName="payload-processing" containerID="cri-o://725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b" gracePeriod=30
Sep 10 21:27:44.680252 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.680194    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:27:44.714389 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.714354    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb289\" (UniqueName: \"kubernetes.io/projected/f40ec1b8-e8ee-4aff-916c-b47891684186-kube-api-access-lb289\") pod \"f40ec1b8-e8ee-4aff-916c-b47891684186\" (UID: \"f40ec1b8-e8ee-4aff-916c-b47891684186\") "
Sep 10 21:27:44.714561 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.714399    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40ec1b8-e8ee-4aff-916c-b47891684186-plugins-config-volume\") pod \"f40ec1b8-e8ee-4aff-916c-b47891684186\" (UID: \"f40ec1b8-e8ee-4aff-916c-b47891684186\") "
Sep 10 21:27:44.714833 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.714805    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40ec1b8-e8ee-4aff-916c-b47891684186-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f40ec1b8-e8ee-4aff-916c-b47891684186" (UID: "f40ec1b8-e8ee-4aff-916c-b47891684186"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:27:44.716612 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.716588    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40ec1b8-e8ee-4aff-916c-b47891684186-kube-api-access-lb289" (OuterVolumeSpecName: "kube-api-access-lb289") pod "f40ec1b8-e8ee-4aff-916c-b47891684186" (UID: "f40ec1b8-e8ee-4aff-916c-b47891684186"). InnerVolumeSpecName "kube-api-access-lb289". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:27:44.815884 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.815854    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb289\" (UniqueName: \"kubernetes.io/projected/f40ec1b8-e8ee-4aff-916c-b47891684186-kube-api-access-lb289\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:27:44.815884 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:44.815878    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f40ec1b8-e8ee-4aff-916c-b47891684186-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:27:45.203872 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.203833    2580 generic.go:356] "Generic (PLEG): container finished" podID="f40ec1b8-e8ee-4aff-916c-b47891684186" containerID="725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b" exitCode=0
Sep 10 21:27:45.203872 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.203875    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"
Sep 10 21:27:45.204157 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.203879    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk" event={"ID":"f40ec1b8-e8ee-4aff-916c-b47891684186","Type":"ContainerDied","Data":"725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b"}
Sep 10 21:27:45.204157 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.204001    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk" event={"ID":"f40ec1b8-e8ee-4aff-916c-b47891684186","Type":"ContainerDied","Data":"49320a9f4c68af0a25e97c7834ba3677154d0950479fb2089374b1df6f19598e"}
Sep 10 21:27:45.204157 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.204029    2580 scope.go:117] "RemoveContainer" containerID="725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b"
Sep 10 21:27:45.215093 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.215073    2580 scope.go:117] "RemoveContainer" containerID="725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b"
Sep 10 21:27:45.215412 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:27:45.215393    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b\": container with ID starting with 725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b not found: ID does not exist" containerID="725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b"
Sep 10 21:27:45.215481 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.215420    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b"} err="failed to get container status \"725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b\": rpc error: code = NotFound desc = could not find container \"725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b\": container with ID starting with 725e1b8e11189594cf54aefcc8fd076c77d292986a087aa9a2dc504d9a3a424b not found: ID does not exist"
Sep 10 21:27:45.224770 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.224748    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"]
Sep 10 21:27:45.228668 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.228646    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a581b5-657db7ccf5-vf6jk"]
Sep 10 21:27:45.519415 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:27:45.519336    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f40ec1b8-e8ee-4aff-916c-b47891684186" path="/var/lib/kubelet/pods/f40ec1b8-e8ee-4aff-916c-b47891684186/volumes"
Sep 10 21:28:20.248981 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.248950    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"]
Sep 10 21:28:20.249546 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.249160    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz" podUID="22a941b7-dc79-42d2-b4b5-e3249453a50c" containerName="maas-api" containerID="cri-o://cfc2f35b16d3abd50acd864aec636f2893a37f537599dcb94c765cd1d9dc52de" gracePeriod=30
Sep 10 21:28:20.308948 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.308915    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"]
Sep 10 21:28:20.309174 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.309148    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn" podUID="e6c41916-3e9a-4bc0-99b7-d13a45c062a7" containerName="payload-processing" containerID="cri-o://cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a" gracePeriod=30
Sep 10 21:28:20.391882 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.391853    2580 generic.go:356] "Generic (PLEG): container finished" podID="22a941b7-dc79-42d2-b4b5-e3249453a50c" containerID="cfc2f35b16d3abd50acd864aec636f2893a37f537599dcb94c765cd1d9dc52de" exitCode=0
Sep 10 21:28:20.392014 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.391923    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz" event={"ID":"22a941b7-dc79-42d2-b4b5-e3249453a50c","Type":"ContainerDied","Data":"cfc2f35b16d3abd50acd864aec636f2893a37f537599dcb94c765cd1d9dc52de"}
Sep 10 21:28:20.569105 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.569075    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:28:20.599880 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.599854    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:28:20.733526 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.733494    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ptvp\" (UniqueName: \"kubernetes.io/projected/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-kube-api-access-5ptvp\") pod \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\" (UID: \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\") "
Sep 10 21:28:20.733709 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.733555    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a941b7-dc79-42d2-b4b5-e3249453a50c-openshift-service-ca\") pod \"22a941b7-dc79-42d2-b4b5-e3249453a50c\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") "
Sep 10 21:28:20.733709 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.733606    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-metrics-certs\") pod \"22a941b7-dc79-42d2-b4b5-e3249453a50c\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") "
Sep 10 21:28:20.733709 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.733642    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-plugins-config-volume\") pod \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\" (UID: \"e6c41916-3e9a-4bc0-99b7-d13a45c062a7\") "
Sep 10 21:28:20.733709 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.733671    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pnj6r\" (UniqueName: \"kubernetes.io/projected/22a941b7-dc79-42d2-b4b5-e3249453a50c-kube-api-access-pnj6r\") pod \"22a941b7-dc79-42d2-b4b5-e3249453a50c\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") "
Sep 10 21:28:20.733709 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.733699    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls\") pod \"22a941b7-dc79-42d2-b4b5-e3249453a50c\" (UID: \"22a941b7-dc79-42d2-b4b5-e3249453a50c\") "
Sep 10 21:28:20.734008 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.733966    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a941b7-dc79-42d2-b4b5-e3249453a50c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "22a941b7-dc79-42d2-b4b5-e3249453a50c" (UID: "22a941b7-dc79-42d2-b4b5-e3249453a50c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:28:20.734080 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.734017    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6c41916-3e9a-4bc0-99b7-d13a45c062a7" (UID: "e6c41916-3e9a-4bc0-99b7-d13a45c062a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:28:20.735925 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.735894    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a941b7-dc79-42d2-b4b5-e3249453a50c-kube-api-access-pnj6r" (OuterVolumeSpecName: "kube-api-access-pnj6r") pod "22a941b7-dc79-42d2-b4b5-e3249453a50c" (UID: "22a941b7-dc79-42d2-b4b5-e3249453a50c"). InnerVolumeSpecName "kube-api-access-pnj6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:28:20.736048 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.735980    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "22a941b7-dc79-42d2-b4b5-e3249453a50c" (UID: "22a941b7-dc79-42d2-b4b5-e3249453a50c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:28:20.736048 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.736035    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "22a941b7-dc79-42d2-b4b5-e3249453a50c" (UID: "22a941b7-dc79-42d2-b4b5-e3249453a50c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:28:20.736162 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.736050    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-kube-api-access-5ptvp" (OuterVolumeSpecName: "kube-api-access-5ptvp") pod "e6c41916-3e9a-4bc0-99b7-d13a45c062a7" (UID: "e6c41916-3e9a-4bc0-99b7-d13a45c062a7"). InnerVolumeSpecName "kube-api-access-5ptvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:28:20.834637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.834544    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ptvp\" (UniqueName: \"kubernetes.io/projected/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-kube-api-access-5ptvp\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:20.834637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.834578    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22a941b7-dc79-42d2-b4b5-e3249453a50c-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:20.834637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.834592    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:20.834637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.834605    2580 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c41916-3e9a-4bc0-99b7-d13a45c062a7-plugins-config-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:20.834637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.834618    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnj6r\" (UniqueName: \"kubernetes.io/projected/22a941b7-dc79-42d2-b4b5-e3249453a50c-kube-api-access-pnj6r\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:20.834637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:20.834631    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22a941b7-dc79-42d2-b4b5-e3249453a50c-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:21.399038 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.399005    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"
Sep 10 21:28:21.399038 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.399035    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz" event={"ID":"22a941b7-dc79-42d2-b4b5-e3249453a50c","Type":"ContainerDied","Data":"afe0506a33c2cc2848e60c57a5c6cd871062b2f944ca4f28c6fd9d5c1de16c34"}
Sep 10 21:28:21.399594 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.399095    2580 scope.go:117] "RemoveContainer" containerID="cfc2f35b16d3abd50acd864aec636f2893a37f537599dcb94c765cd1d9dc52de"
Sep 10 21:28:21.400637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.400611    2580 generic.go:356] "Generic (PLEG): container finished" podID="e6c41916-3e9a-4bc0-99b7-d13a45c062a7" containerID="cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a" exitCode=0
Sep 10 21:28:21.400754 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.400654    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"
Sep 10 21:28:21.400754 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.400663    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn" event={"ID":"e6c41916-3e9a-4bc0-99b7-d13a45c062a7","Type":"ContainerDied","Data":"cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a"}
Sep 10 21:28:21.400754 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.400698    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn" event={"ID":"e6c41916-3e9a-4bc0-99b7-d13a45c062a7","Type":"ContainerDied","Data":"6784efb5250f7c840acd88f7ae17f5deef7801a9052e7d788fcfe477065e685e"}
Sep 10 21:28:21.412090 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.412072    2580 scope.go:117] "RemoveContainer" containerID="cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a"
Sep 10 21:28:21.420890 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.420873    2580 scope.go:117] "RemoveContainer" containerID="cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a"
Sep 10 21:28:21.421187 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:28:21.421164    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a\": container with ID starting with cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a not found: ID does not exist" containerID="cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a"
Sep 10 21:28:21.421238 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.421201    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a"} err="failed to get container status \"cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a\": rpc error: code = NotFound desc = could not find container \"cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a\": container with ID starting with cd057681732d1fadeba1ca80e74659ac3f47e9b0a8c321d266e5cd7cc645884a not found: ID does not exist"
Sep 10 21:28:21.427097 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.427077    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"]
Sep 10 21:28:21.436307 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.436262    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-16bbec-df9974d4c-pdnqn"]
Sep 10 21:28:21.446095 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.446077    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"]
Sep 10 21:28:21.450804 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.450776    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-16bbec-774c4bbdc6-js6vz"]
Sep 10 21:28:21.519433 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.519358    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a941b7-dc79-42d2-b4b5-e3249453a50c" path="/var/lib/kubelet/pods/22a941b7-dc79-42d2-b4b5-e3249453a50c/volumes"
Sep 10 21:28:21.519703 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:21.519689    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c41916-3e9a-4bc0-99b7-d13a45c062a7" path="/var/lib/kubelet/pods/e6c41916-3e9a-4bc0-99b7-d13a45c062a7/volumes"
Sep 10 21:28:27.114691 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:27.114613    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"]
Sep 10 21:28:27.115153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:27.114833    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy" containerID="cri-o://dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04" gracePeriod=30
Sep 10 21:28:29.795711 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:29.795672    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84497f567c-l4ddm"]
Sep 10 21:28:29.796150 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:29.795934    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84497f567c-l4ddm" podUID="c65b4183-378d-4533-91a1-7ed2824994d7" containerName="manager" containerID="cri-o://0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84" gracePeriod=30
Sep 10 21:28:30.042997 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.042970    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:28:30.220476 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.220442    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs\") pod \"c65b4183-378d-4533-91a1-7ed2824994d7\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") "
Sep 10 21:28:30.220646 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.220531    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cjmk\" (UniqueName: \"kubernetes.io/projected/c65b4183-378d-4533-91a1-7ed2824994d7-kube-api-access-8cjmk\") pod \"c65b4183-378d-4533-91a1-7ed2824994d7\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") "
Sep 10 21:28:30.220646 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.220566    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-webhook-cert\") pod \"c65b4183-378d-4533-91a1-7ed2824994d7\" (UID: \"c65b4183-378d-4533-91a1-7ed2824994d7\") "
Sep 10 21:28:30.222650 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.222611    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c65b4183-378d-4533-91a1-7ed2824994d7" (UID: "c65b4183-378d-4533-91a1-7ed2824994d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:28:30.222771 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.222727    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c65b4183-378d-4533-91a1-7ed2824994d7" (UID: "c65b4183-378d-4533-91a1-7ed2824994d7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:28:30.222771 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.222732    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65b4183-378d-4533-91a1-7ed2824994d7-kube-api-access-8cjmk" (OuterVolumeSpecName: "kube-api-access-8cjmk") pod "c65b4183-378d-4533-91a1-7ed2824994d7" (UID: "c65b4183-378d-4533-91a1-7ed2824994d7"). InnerVolumeSpecName "kube-api-access-8cjmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:28:30.321321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.321293    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cjmk\" (UniqueName: \"kubernetes.io/projected/c65b4183-378d-4533-91a1-7ed2824994d7-kube-api-access-8cjmk\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:30.321321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.321320    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-webhook-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:30.321321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.321329    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c65b4183-378d-4533-91a1-7ed2824994d7-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:30.454237 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.454207    2580 generic.go:356] "Generic (PLEG): container finished" podID="c65b4183-378d-4533-91a1-7ed2824994d7" containerID="0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84" exitCode=0
Sep 10 21:28:30.454403 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.454257    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-l4ddm"
Sep 10 21:28:30.454403 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.454308    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-l4ddm" event={"ID":"c65b4183-378d-4533-91a1-7ed2824994d7","Type":"ContainerDied","Data":"0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84"}
Sep 10 21:28:30.454403 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.454357    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-l4ddm" event={"ID":"c65b4183-378d-4533-91a1-7ed2824994d7","Type":"ContainerDied","Data":"6929bb72121227b7b429850f2fcec6bc9a664b6eefc9f333535de58e9b419e27"}
Sep 10 21:28:30.454403 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.454379    2580 scope.go:117] "RemoveContainer" containerID="0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84"
Sep 10 21:28:30.464278 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.464259    2580 scope.go:117] "RemoveContainer" containerID="0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84"
Sep 10 21:28:30.464572 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:28:30.464554    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84\": container with ID starting with 0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84 not found: ID does not exist" containerID="0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84"
Sep 10 21:28:30.464637 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.464580    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84"} err="failed to get container status \"0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84\": rpc error: code = NotFound desc = could not find container \"0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84\": container with ID starting with 0096447c1b5a2531067c55f51bfb85663cf1e9bbe080d2e95321fa96862c1c84 not found: ID does not exist"
Sep 10 21:28:30.477289 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.477231    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84497f567c-l4ddm"]
Sep 10 21:28:30.480919 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:30.480899    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84497f567c-l4ddm"]
Sep 10 21:28:31.518704 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:31.518665    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c65b4183-378d-4533-91a1-7ed2824994d7" path="/var/lib/kubelet/pods/c65b4183-378d-4533-91a1-7ed2824994d7/volumes"
Sep 10 21:28:32.361223 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.361197    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:28:32.469006 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.468927    2580 generic.go:356] "Generic (PLEG): container finished" podID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerID="dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04" exitCode=0
Sep 10 21:28:32.469006 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.468965    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" event={"ID":"8497ce30-4b56-410a-967f-4b09b7629e8f","Type":"ContainerDied","Data":"dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04"}
Sep 10 21:28:32.469006 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.468974    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"
Sep 10 21:28:32.469006 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.469003    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf" event={"ID":"8497ce30-4b56-410a-967f-4b09b7629e8f","Type":"ContainerDied","Data":"72b4108742c1367e1e57bad78ab333827efdbd8d6da8d6bc4a43aed84091664c"}
Sep 10 21:28:32.469359 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.469023    2580 scope.go:117] "RemoveContainer" containerID="dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04"
Sep 10 21:28:32.477953 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.477937    2580 scope.go:117] "RemoveContainer" containerID="02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37"
Sep 10 21:28:32.486411 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.486396    2580 scope.go:117] "RemoveContainer" containerID="dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04"
Sep 10 21:28:32.486687 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:28:32.486667    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04\": container with ID starting with dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04 not found: ID does not exist" containerID="dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04"
Sep 10 21:28:32.486760 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.486699    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04"} err="failed to get container status \"dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04\": rpc error: code = NotFound desc = could not find container \"dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04\": container with ID starting with dd0044883cec0ef344e47ef0da9c5a9f906d9b672310d0eb7fff933810aa4a04 not found: ID does not exist"
Sep 10 21:28:32.486760 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.486723    2580 scope.go:117] "RemoveContainer" containerID="02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37"
Sep 10 21:28:32.486990 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:28:32.486970    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37\": container with ID starting with 02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37 not found: ID does not exist" containerID="02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37"
Sep 10 21:28:32.487027 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.487000    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37"} err="failed to get container status \"02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37\": rpc error: code = NotFound desc = could not find container \"02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37\": container with ID starting with 02317d31ba628117f10cb4c779209ef87ad0118faaa43f593e66adb612f4bf37 not found: ID does not exist"
Sep 10 21:28:32.541819 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.541790    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-certs\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.541832    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-socket\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.541899    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cs5pg\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-kube-api-access-cs5pg\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.541988    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-podinfo\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542022    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-token\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542046    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:32.542194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542068    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497ce30-4b56-410a-967f-4b09b7629e8f-istiod-ca-cert\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542146    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-data\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542184    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-credential-socket\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542184    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542220    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-envoy\") pod \"8497ce30-4b56-410a-967f-4b09b7629e8f\" (UID: \"8497ce30-4b56-410a-967f-4b09b7629e8f\") "
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542383    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-data" (OuterVolumeSpecName: "istio-data") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542422    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8497ce30-4b56-410a-967f-4b09b7629e8f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542551    2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-data\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542635    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542570    2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542662    2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-workload-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.542853 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.542676    2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8497ce30-4b56-410a-967f-4b09b7629e8f-istiod-ca-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.544319 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.544297    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:28:32.544428 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.544378    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-kube-api-access-cs5pg" (OuterVolumeSpecName: "kube-api-access-cs5pg") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "kube-api-access-cs5pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:28:32.544667 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.544652    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 21:28:32.544742 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.544663    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-token" (OuterVolumeSpecName: "istio-token") pod "8497ce30-4b56-410a-967f-4b09b7629e8f" (UID: "8497ce30-4b56-410a-967f-4b09b7629e8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:28:32.643178 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.643154    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs5pg\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-kube-api-access-cs5pg\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.643178 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.643177    2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-podinfo\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.643327 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.643187    2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-token\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.643327 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.643196    2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-credential-socket\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.643327 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.643204    2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8497ce30-4b56-410a-967f-4b09b7629e8f-istio-envoy\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:28:32.792470 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.792439    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"]
Sep 10 21:28:32.796506 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:32.796482    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-16bbec-openshift-default-7bcf675484-d47xf"]
Sep 10 21:28:33.519086 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:28:33.519043    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" path="/var/lib/kubelet/pods/8497ce30-4b56-410a-967f-4b09b7629e8f/volumes"
Sep 10 21:29:31.329591 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.329554    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84497f567c-pbpqz"]
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330045    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330065    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330082    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c41916-3e9a-4bc0-99b7-d13a45c062a7" containerName="payload-processing"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330091    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c41916-3e9a-4bc0-99b7-d13a45c062a7" containerName="payload-processing"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330102    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a941b7-dc79-42d2-b4b5-e3249453a50c" containerName="maas-api"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330111    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a941b7-dc79-42d2-b4b5-e3249453a50c" containerName="maas-api"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330144    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" containerName="maas-api"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330153    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" containerName="maas-api"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330163    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f40ec1b8-e8ee-4aff-916c-b47891684186" containerName="payload-processing"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330170    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f40ec1b8-e8ee-4aff-916c-b47891684186" containerName="payload-processing"
Sep 10 21:29:31.330180 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330183    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dc075b7-f237-4e46-99f9-cfdee083af49" containerName="payload-processing"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330192    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dc075b7-f237-4e46-99f9-cfdee083af49" containerName="payload-processing"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330209    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330216    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330225    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe2b173d-141f-43b9-a43d-e724f9dd6c97" containerName="maas-api"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330233    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe2b173d-141f-43b9-a43d-e724f9dd6c97" containerName="maas-api"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330249    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c65b4183-378d-4533-91a1-7ed2824994d7" containerName="manager"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330256    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c65b4183-378d-4533-91a1-7ed2824994d7" containerName="manager"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330356    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a941b7-dc79-42d2-b4b5-e3249453a50c" containerName="maas-api"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330368    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c65b4183-378d-4533-91a1-7ed2824994d7" containerName="manager"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330377    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dc075b7-f237-4e46-99f9-cfdee083af49" containerName="payload-processing"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330385    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="898ca11d-13b8-4ff3-be94-ee9cd72eaa9e" containerName="maas-api"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330396    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f40ec1b8-e8ee-4aff-916c-b47891684186" containerName="payload-processing"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330405    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330414    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8497ce30-4b56-410a-967f-4b09b7629e8f" containerName="istio-proxy"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330422    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe2b173d-141f-43b9-a43d-e724f9dd6c97" containerName="maas-api"
Sep 10 21:29:31.330748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.330431    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c41916-3e9a-4bc0-99b7-d13a45c062a7" containerName="payload-processing"
Sep 10 21:29:31.333577 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.333556    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.336663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.336627    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 21:29:31.336779 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.336629    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 21:29:31.336779 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.336629    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 21:29:31.336779 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.336635    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6h5kw\""
Sep 10 21:29:31.346185 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.346166    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84497f567c-pbpqz"]
Sep 10 21:29:31.448426 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.448399    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-metrics-certs\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.448589 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.448430    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpwj\" (UniqueName: \"kubernetes.io/projected/3ba2844a-408c-438a-a0a4-78589d8d51ce-kube-api-access-pwpwj\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.448589 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.448475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-webhook-cert\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.549825 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.549793    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-webhook-cert\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.550000 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.549840    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-metrics-certs\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.550000 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.549955    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpwj\" (UniqueName: \"kubernetes.io/projected/3ba2844a-408c-438a-a0a4-78589d8d51ce-kube-api-access-pwpwj\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.552334 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.552308    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-metrics-certs\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.552460 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.552386    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-webhook-cert\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.558395 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.558372    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpwj\" (UniqueName: \"kubernetes.io/projected/3ba2844a-408c-438a-a0a4-78589d8d51ce-kube-api-access-pwpwj\") pod \"maas-controller-84497f567c-pbpqz\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") " pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.644461 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.644416    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:31.916736 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.916707    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84497f567c-pbpqz"]
Sep 10 21:29:31.919260 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:29:31.919231    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba2844a_408c_438a_a0a4_78589d8d51ce.slice/crio-d592741e0d4a87174ce1c354509d00e62a54b6e914df854992ab3b8eb9d626b2 WatchSource:0}: Error finding container d592741e0d4a87174ce1c354509d00e62a54b6e914df854992ab3b8eb9d626b2: Status 404 returned error can't find the container with id d592741e0d4a87174ce1c354509d00e62a54b6e914df854992ab3b8eb9d626b2
Sep 10 21:29:31.920658 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:31.920640    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:29:32.796203 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:32.796072    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-pbpqz" event={"ID":"3ba2844a-408c-438a-a0a4-78589d8d51ce","Type":"ContainerStarted","Data":"ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe"}
Sep 10 21:29:32.796203 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:32.796155    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-pbpqz" event={"ID":"3ba2844a-408c-438a-a0a4-78589d8d51ce","Type":"ContainerStarted","Data":"d592741e0d4a87174ce1c354509d00e62a54b6e914df854992ab3b8eb9d626b2"}
Sep 10 21:29:32.796203 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:32.796188    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:32.842332 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:32.842276    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84497f567c-pbpqz" podStartSLOduration=1.308121075 podStartE2EDuration="1.842256256s" totalImagesPullingTime="534.135181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:29:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:29:32.842061218 +0000 UTC m=+1859.844307577" watchObservedRunningTime="2026-09-10 21:29:32.842256256 +0000 UTC m=+1859.844502604"
Sep 10 21:29:43.803521 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:43.803489    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:53.223324 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.223286    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84497f567c-pbpqz"]
Sep 10 21:29:53.223821 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.223576    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84497f567c-pbpqz" podUID="3ba2844a-408c-438a-a0a4-78589d8d51ce" containerName="manager" containerID="cri-o://ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe" gracePeriod=30
Sep 10 21:29:53.475739 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.475676    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:53.531781 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.531754    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-webhook-cert\") pod \"3ba2844a-408c-438a-a0a4-78589d8d51ce\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") "
Sep 10 21:29:53.531948 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.531858    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwpwj\" (UniqueName: \"kubernetes.io/projected/3ba2844a-408c-438a-a0a4-78589d8d51ce-kube-api-access-pwpwj\") pod \"3ba2844a-408c-438a-a0a4-78589d8d51ce\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") "
Sep 10 21:29:53.531948 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.531900    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-metrics-certs\") pod \"3ba2844a-408c-438a-a0a4-78589d8d51ce\" (UID: \"3ba2844a-408c-438a-a0a4-78589d8d51ce\") "
Sep 10 21:29:53.533968 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.533938    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "3ba2844a-408c-438a-a0a4-78589d8d51ce" (UID: "3ba2844a-408c-438a-a0a4-78589d8d51ce"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:29:53.534168 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.534039    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3ba2844a-408c-438a-a0a4-78589d8d51ce" (UID: "3ba2844a-408c-438a-a0a4-78589d8d51ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:29:53.534268 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.534244    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba2844a-408c-438a-a0a4-78589d8d51ce-kube-api-access-pwpwj" (OuterVolumeSpecName: "kube-api-access-pwpwj") pod "3ba2844a-408c-438a-a0a4-78589d8d51ce" (UID: "3ba2844a-408c-438a-a0a4-78589d8d51ce"). InnerVolumeSpecName "kube-api-access-pwpwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:29:53.633599 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.633564    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpwj\" (UniqueName: \"kubernetes.io/projected/3ba2844a-408c-438a-a0a4-78589d8d51ce-kube-api-access-pwpwj\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:29:53.633834 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.633810    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:29:53.633834 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.633837    2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ba2844a-408c-438a-a0a4-78589d8d51ce-webhook-cert\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:29:53.904462 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.904433    2580 generic.go:356] "Generic (PLEG): container finished" podID="3ba2844a-408c-438a-a0a4-78589d8d51ce" containerID="ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe" exitCode=0
Sep 10 21:29:53.904642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.904476    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-pbpqz"
Sep 10 21:29:53.904642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.904523    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-pbpqz" event={"ID":"3ba2844a-408c-438a-a0a4-78589d8d51ce","Type":"ContainerDied","Data":"ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe"}
Sep 10 21:29:53.904642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.904573    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-pbpqz" event={"ID":"3ba2844a-408c-438a-a0a4-78589d8d51ce","Type":"ContainerDied","Data":"d592741e0d4a87174ce1c354509d00e62a54b6e914df854992ab3b8eb9d626b2"}
Sep 10 21:29:53.904642 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.904595    2580 scope.go:117] "RemoveContainer" containerID="ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe"
Sep 10 21:29:53.913974 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.913958    2580 scope.go:117] "RemoveContainer" containerID="ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe"
Sep 10 21:29:53.914337 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:29:53.914317    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe\": container with ID starting with ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe not found: ID does not exist" containerID="ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe"
Sep 10 21:29:53.914412 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.914342    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe"} err="failed to get container status \"ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe\": rpc error: code = NotFound desc = could not find container \"ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe\": container with ID starting with ab6a3c121f08c17150d9619a934dfd1c84a2d4144a60465661d4ab5e046aeafe not found: ID does not exist"
Sep 10 21:29:53.934805 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.934776    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84497f567c-pbpqz"]
Sep 10 21:29:53.941369 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:53.939272    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84497f567c-pbpqz"]
Sep 10 21:29:54.793908 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.793828    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84497f567c-2m9rg"]
Sep 10 21:29:54.794269 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.794218    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba2844a-408c-438a-a0a4-78589d8d51ce" containerName="manager"
Sep 10 21:29:54.794269 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.794229    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba2844a-408c-438a-a0a4-78589d8d51ce" containerName="manager"
Sep 10 21:29:54.794340 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.794298    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba2844a-408c-438a-a0a4-78589d8d51ce" containerName="manager"
Sep 10 21:29:54.798225 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.798208    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:54.801153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.801111    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 21:29:54.801262 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.801184    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6h5kw\""
Sep 10 21:29:54.801262 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.801186    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 21:29:54.801372 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.801192    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 21:29:54.809188 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.809165    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84497f567c-2m9rg"]
Sep 10 21:29:54.945775 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.945748    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-metrics-certs\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:54.945926 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.945788    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hq6s\" (UniqueName: \"kubernetes.io/projected/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-kube-api-access-6hq6s\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:54.945926 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:54.945907    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-webhook-cert\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.046518 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.046415    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-webhook-cert\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.046518 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.046469    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-metrics-certs\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.046518 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.046507    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hq6s\" (UniqueName: \"kubernetes.io/projected/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-kube-api-access-6hq6s\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.049013 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.048988    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-metrics-certs\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.049168 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.049147    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-webhook-cert\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.058506 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.058485    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hq6s\" (UniqueName: \"kubernetes.io/projected/26d9dc4f-9497-47d9-9d3a-2db00d1d3882-kube-api-access-6hq6s\") pod \"maas-controller-84497f567c-2m9rg\" (UID: \"26d9dc4f-9497-47d9-9d3a-2db00d1d3882\") " pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.108844 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.108813    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.383987 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.383962    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84497f567c-2m9rg"]
Sep 10 21:29:55.385937 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:29:55.385890    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26d9dc4f_9497_47d9_9d3a_2db00d1d3882.slice/crio-6d8678b35c560aff17e7865887f279f2487e054ea5b2c3c22b0e460a12d11dfa WatchSource:0}: Error finding container 6d8678b35c560aff17e7865887f279f2487e054ea5b2c3c22b0e460a12d11dfa: Status 404 returned error can't find the container with id 6d8678b35c560aff17e7865887f279f2487e054ea5b2c3c22b0e460a12d11dfa
Sep 10 21:29:55.519252 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.519214    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba2844a-408c-438a-a0a4-78589d8d51ce" path="/var/lib/kubelet/pods/3ba2844a-408c-438a-a0a4-78589d8d51ce/volumes"
Sep 10 21:29:55.919500 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.919444    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-2m9rg" event={"ID":"26d9dc4f-9497-47d9-9d3a-2db00d1d3882","Type":"ContainerStarted","Data":"8b5b014726f2b3036600d142858f32c848bf42820abf915ba40f17ff13ffee12"}
Sep 10 21:29:55.919996 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.919503    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84497f567c-2m9rg" event={"ID":"26d9dc4f-9497-47d9-9d3a-2db00d1d3882","Type":"ContainerStarted","Data":"6d8678b35c560aff17e7865887f279f2487e054ea5b2c3c22b0e460a12d11dfa"}
Sep 10 21:29:55.919996 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.919532    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:29:55.940429 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:29:55.940369    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84497f567c-2m9rg" podStartSLOduration=1.521297923 podStartE2EDuration="1.940353694s" totalImagesPullingTime="419.055771ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:29:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:29:55.93792915 +0000 UTC m=+1882.940175508" watchObservedRunningTime="2026-09-10 21:29:55.940353694 +0000 UTC m=+1882.942600091"
Sep 10 21:30:06.927350 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:30:06.927321    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84497f567c-2m9rg"
Sep 10 21:31:04.898131 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:04.898081    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"]
Sep 10 21:31:04.898714 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:04.898293    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp" podUID="5c70afef-d251-4df2-ac24-595bf202ff6d" containerName="manager" containerID="cri-o://00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869" gracePeriod=10
Sep 10 21:31:06.255065 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.255040    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:31:06.293316 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.293287    2580 generic.go:356] "Generic (PLEG): container finished" podID="5c70afef-d251-4df2-ac24-595bf202ff6d" containerID="00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869" exitCode=0
Sep 10 21:31:06.293477 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.293329    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp" event={"ID":"5c70afef-d251-4df2-ac24-595bf202ff6d","Type":"ContainerDied","Data":"00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869"}
Sep 10 21:31:06.293477 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.293365    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"
Sep 10 21:31:06.293477 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.293412    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp" event={"ID":"5c70afef-d251-4df2-ac24-595bf202ff6d","Type":"ContainerDied","Data":"d40a71e42f6d7144a30cce5317bf842a36e1d9f3a9f2c31ccb8682e42e45ffc9"}
Sep 10 21:31:06.293477 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.293446    2580 scope.go:117] "RemoveContainer" containerID="00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869"
Sep 10 21:31:06.303321 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.303291    2580 scope.go:117] "RemoveContainer" containerID="00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869"
Sep 10 21:31:06.303619 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:31:06.303599    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869\": container with ID starting with 00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869 not found: ID does not exist" containerID="00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869"
Sep 10 21:31:06.303730 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.303638    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869"} err="failed to get container status \"00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869\": rpc error: code = NotFound desc = could not find container \"00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869\": container with ID starting with 00534e058761d9381b031d2923e59eca913c5d1d4c0d9e34e1a56ab0fe65a869 not found: ID does not exist"
Sep 10 21:31:06.346153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.346077    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c70afef-d251-4df2-ac24-595bf202ff6d-extensions-socket-volume\") pod \"5c70afef-d251-4df2-ac24-595bf202ff6d\" (UID: \"5c70afef-d251-4df2-ac24-595bf202ff6d\") "
Sep 10 21:31:06.346153 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.346113    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl79j\" (UniqueName: \"kubernetes.io/projected/5c70afef-d251-4df2-ac24-595bf202ff6d-kube-api-access-kl79j\") pod \"5c70afef-d251-4df2-ac24-595bf202ff6d\" (UID: \"5c70afef-d251-4df2-ac24-595bf202ff6d\") "
Sep 10 21:31:06.346425 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.346402    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c70afef-d251-4df2-ac24-595bf202ff6d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5c70afef-d251-4df2-ac24-595bf202ff6d" (UID: "5c70afef-d251-4df2-ac24-595bf202ff6d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:31:06.348159 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.348140    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c70afef-d251-4df2-ac24-595bf202ff6d-kube-api-access-kl79j" (OuterVolumeSpecName: "kube-api-access-kl79j") pod "5c70afef-d251-4df2-ac24-595bf202ff6d" (UID: "5c70afef-d251-4df2-ac24-595bf202ff6d"). InnerVolumeSpecName "kube-api-access-kl79j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:31:06.447565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.447541    2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c70afef-d251-4df2-ac24-595bf202ff6d-extensions-socket-volume\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:31:06.447565 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.447564    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl79j\" (UniqueName: \"kubernetes.io/projected/5c70afef-d251-4df2-ac24-595bf202ff6d-kube-api-access-kl79j\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:31:06.615795 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.615720    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"]
Sep 10 21:31:06.618411 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:06.618385    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-bxtfp"]
Sep 10 21:31:07.518969 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:31:07.518936    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c70afef-d251-4df2-ac24-595bf202ff6d" path="/var/lib/kubelet/pods/5c70afef-d251-4df2-ac24-595bf202ff6d/volumes"
Sep 10 21:32:05.637489 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.637460    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"]
Sep 10 21:32:05.637937 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.637818    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c70afef-d251-4df2-ac24-595bf202ff6d" containerName="manager"
Sep 10 21:32:05.637937 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.637830    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c70afef-d251-4df2-ac24-595bf202ff6d" containerName="manager"
Sep 10 21:32:05.637937 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.637886    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c70afef-d251-4df2-ac24-595bf202ff6d" containerName="manager"
Sep 10 21:32:05.641012 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.640996    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:05.644250 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.644230    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9dkn7\""
Sep 10 21:32:05.665163 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.665115    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"]
Sep 10 21:32:05.733506 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.733475    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgjv5\" (UID: \"69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:05.733641 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.733527    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp9ng\" (UniqueName: \"kubernetes.io/projected/69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f-kube-api-access-qp9ng\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgjv5\" (UID: \"69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:05.834928 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.834897    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgjv5\" (UID: \"69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:05.835041 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.834954    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp9ng\" (UniqueName: \"kubernetes.io/projected/69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f-kube-api-access-qp9ng\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgjv5\" (UID: \"69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:05.835347 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.835327    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgjv5\" (UID: \"69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:05.843547 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.843521    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp9ng\" (UniqueName: \"kubernetes.io/projected/69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f-kube-api-access-qp9ng\") pod \"kuadrant-operator-controller-manager-5769f99d9f-mgjv5\" (UID: \"69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:05.950269 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:05.950195    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:06.228609 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:06.228578    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"]
Sep 10 21:32:06.230135 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:32:06.230095    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69aaf1ba_f13d_4ac9_a703_ab278a7ffe6f.slice/crio-ba2f03867afdae53bfcce502e3807e480170f74f6aca7843dbebb582fe0a5f20 WatchSource:0}: Error finding container ba2f03867afdae53bfcce502e3807e480170f74f6aca7843dbebb582fe0a5f20: Status 404 returned error can't find the container with id ba2f03867afdae53bfcce502e3807e480170f74f6aca7843dbebb582fe0a5f20
Sep 10 21:32:06.608400 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:06.608299    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5" event={"ID":"69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f","Type":"ContainerStarted","Data":"33b6bf321d0ebd5c038c4ce90c62d5e25a5291bc7db00ba7fbcd85bc63338bd3"}
Sep 10 21:32:06.608400 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:06.608359    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:32:06.608400 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:06.608378    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5" event={"ID":"69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f","Type":"ContainerStarted","Data":"ba2f03867afdae53bfcce502e3807e480170f74f6aca7843dbebb582fe0a5f20"}
Sep 10 21:32:06.633089 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:06.633038    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5" podStartSLOduration=1.633022588 podStartE2EDuration="1.633022588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:32:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:32:06.631104355 +0000 UTC m=+2013.633350712" watchObservedRunningTime="2026-09-10 21:32:06.633022588 +0000 UTC m=+2013.635268937"
Sep 10 21:32:17.616213 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:32:17.616186    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-mgjv5"
Sep 10 21:33:07.462479 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.462445    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"]
Sep 10 21:33:07.466244 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.466222    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.468595 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.468574    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-eae8a67b-kserve-self-signed-certs\""
Sep 10 21:33:07.478308 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.478284    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"]
Sep 10 21:33:07.657293 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.657259    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-model-cache\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.657293 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.657302    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tls-certs\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.657535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.657351    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx8kg\" (UniqueName: \"kubernetes.io/projected/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-kube-api-access-nx8kg\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.657535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.657396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-home\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.657535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.657431    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-dshm\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.657535 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.657459    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tmp-dir\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.758626 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.758536    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tls-certs\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.758626 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.758588    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx8kg\" (UniqueName: \"kubernetes.io/projected/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-kube-api-access-nx8kg\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.758626 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.758615    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-home\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.758899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.758640    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-dshm\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.758899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.758671    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tmp-dir\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.758899 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.758777    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-model-cache\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.759069 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.759046    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-home\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.759163 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.759105    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-model-cache\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.759215 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.759160    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tmp-dir\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.761150 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.761114    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-dshm\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.761411 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.761395    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tls-certs\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.766736 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.766718    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx8kg\" (UniqueName: \"kubernetes.io/projected/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-kube-api-access-nx8kg\") pod \"e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") " pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:07.775639 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:07.775619    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:08.054292 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:08.054263    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"]
Sep 10 21:33:08.055804 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:33:08.055758    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f50f3b1_5d57_4e9c_9a84_8605aef5a503.slice/crio-5ace84609be884d608c794ffcba537c46d1597e872031ae72a5f45c4a630fc61 WatchSource:0}: Error finding container 5ace84609be884d608c794ffcba537c46d1597e872031ae72a5f45c4a630fc61: Status 404 returned error can't find the container with id 5ace84609be884d608c794ffcba537c46d1597e872031ae72a5f45c4a630fc61
Sep 10 21:33:08.935586 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:08.935551    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz" event={"ID":"6f50f3b1-5d57-4e9c-9a84-8605aef5a503","Type":"ContainerStarted","Data":"583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3"}
Sep 10 21:33:08.935586 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:08.935597    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz" event={"ID":"6f50f3b1-5d57-4e9c-9a84-8605aef5a503","Type":"ContainerStarted","Data":"5ace84609be884d608c794ffcba537c46d1597e872031ae72a5f45c4a630fc61"}
Sep 10 21:33:08.952517 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:08.952471    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz" podStartSLOduration=1.952457505 podStartE2EDuration="1.952457505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:33:08.951524719 +0000 UTC m=+2075.953771090" watchObservedRunningTime="2026-09-10 21:33:08.952457505 +0000 UTC m=+2075.954703851"
Sep 10 21:33:17.776378 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:17.776334    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:17.776897 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:17.776397    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:17.792429 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:17.792402    2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:17.995021 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:17.994993    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:42.815916 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:42.815878    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"]
Sep 10 21:33:57.836460 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:57.836413    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz" podUID="6f50f3b1-5d57-4e9c-9a84-8605aef5a503" containerName="main" containerID="cri-o://583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3" gracePeriod=45
Sep 10 21:33:57.984997 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:57.984961    2580 prober.go:120] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz" podUID="6f50f3b1-5d57-4e9c-9a84-8605aef5a503" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": dial tcp 10.133.0.90:8000: connect: connection refused"
Sep 10 21:33:58.083241 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.083214    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:58.180537 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.180510    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tls-certs\") pod \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") "
Sep 10 21:33:58.180720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.180572    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-model-cache\") pod \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") "
Sep 10 21:33:58.180720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.180605    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tmp-dir\") pod \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") "
Sep 10 21:33:58.180720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.180649    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-dshm\") pod \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") "
Sep 10 21:33:58.180720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.180678    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx8kg\" (UniqueName: \"kubernetes.io/projected/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-kube-api-access-nx8kg\") pod \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") "
Sep 10 21:33:58.180720 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.180717    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-home\") pod \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\" (UID: \"6f50f3b1-5d57-4e9c-9a84-8605aef5a503\") "
Sep 10 21:33:58.180998 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.180907    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f50f3b1-5d57-4e9c-9a84-8605aef5a503" (UID: "6f50f3b1-5d57-4e9c-9a84-8605aef5a503"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:33:58.180998 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.180917    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f50f3b1-5d57-4e9c-9a84-8605aef5a503" (UID: "6f50f3b1-5d57-4e9c-9a84-8605aef5a503"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:33:58.181111 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.181095    2580 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-model-cache\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:33:58.181192 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.181114    2580 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tmp-dir\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:33:58.181192 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.181138    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-home" (OuterVolumeSpecName: "home") pod "6f50f3b1-5d57-4e9c-9a84-8605aef5a503" (UID: "6f50f3b1-5d57-4e9c-9a84-8605aef5a503"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:33:58.183020 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.182996    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-dshm" (OuterVolumeSpecName: "dshm") pod "6f50f3b1-5d57-4e9c-9a84-8605aef5a503" (UID: "6f50f3b1-5d57-4e9c-9a84-8605aef5a503"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 21:33:58.183020 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.183001    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f50f3b1-5d57-4e9c-9a84-8605aef5a503" (UID: "6f50f3b1-5d57-4e9c-9a84-8605aef5a503"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:33:58.183193 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.183037    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-kube-api-access-nx8kg" (OuterVolumeSpecName: "kube-api-access-nx8kg") pod "6f50f3b1-5d57-4e9c-9a84-8605aef5a503" (UID: "6f50f3b1-5d57-4e9c-9a84-8605aef5a503"). InnerVolumeSpecName "kube-api-access-nx8kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:33:58.197969 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.197946    2580 generic.go:356] "Generic (PLEG): container finished" podID="6f50f3b1-5d57-4e9c-9a84-8605aef5a503" containerID="583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3" exitCode=0
Sep 10 21:33:58.198078 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.197978    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz" event={"ID":"6f50f3b1-5d57-4e9c-9a84-8605aef5a503","Type":"ContainerDied","Data":"583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3"}
Sep 10 21:33:58.198078 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.197993    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"
Sep 10 21:33:58.198078 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.198005    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz" event={"ID":"6f50f3b1-5d57-4e9c-9a84-8605aef5a503","Type":"ContainerDied","Data":"5ace84609be884d608c794ffcba537c46d1597e872031ae72a5f45c4a630fc61"}
Sep 10 21:33:58.198078 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.198021    2580 scope.go:117] "RemoveContainer" containerID="583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3"
Sep 10 21:33:58.207227 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.207212    2580 scope.go:117] "RemoveContainer" containerID="583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3"
Sep 10 21:33:58.207467 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:33:58.207451    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3\": container with ID starting with 583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3 not found: ID does not exist" containerID="583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3"
Sep 10 21:33:58.207518 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.207475    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3"} err="failed to get container status \"583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3\": rpc error: code = NotFound desc = could not find container \"583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3\": container with ID starting with 583247e7715c02d9536ffaaa9be6193a3d7ada53c6b7b28c9126085149c905e3 not found: ID does not exist"
Sep 10 21:33:58.222567 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.222548    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"]
Sep 10 21:33:58.228329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.228310    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-eae8a67b-kserve-bb9fbc679-h84gz"]
Sep 10 21:33:58.281686 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.281667    2580 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-tls-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:33:58.281686 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.281684    2580 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-dshm\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:33:58.281802 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.281694    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx8kg\" (UniqueName: \"kubernetes.io/projected/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-kube-api-access-nx8kg\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:33:58.281802 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:58.281703    2580 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f50f3b1-5d57-4e9c-9a84-8605aef5a503-home\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:33:59.518758 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:33:59.518726    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f50f3b1-5d57-4e9c-9a84-8605aef5a503" path="/var/lib/kubelet/pods/6f50f3b1-5d57-4e9c-9a84-8605aef5a503/volumes"
Sep 10 21:34:06.167881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.167843    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"]
Sep 10 21:34:06.168329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.168231    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f50f3b1-5d57-4e9c-9a84-8605aef5a503" containerName="main"
Sep 10 21:34:06.168329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.168249    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f50f3b1-5d57-4e9c-9a84-8605aef5a503" containerName="main"
Sep 10 21:34:06.168329 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.168326    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f50f3b1-5d57-4e9c-9a84-8605aef5a503" containerName="main"
Sep 10 21:34:06.172944 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.172919    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.183643 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.183541    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"]
Sep 10 21:34:06.349292 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.349260    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbzf8\" (UniqueName: \"kubernetes.io/projected/78aa99a1-a611-4e29-98d0-5095769e04b8-kube-api-access-bbzf8\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.349443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.349327    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78aa99a1-a611-4e29-98d0-5095769e04b8-maas-api-tls\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.349443 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.349396    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78aa99a1-a611-4e29-98d0-5095769e04b8-metrics-certs\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.349537 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.349491    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78aa99a1-a611-4e29-98d0-5095769e04b8-openshift-service-ca\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.450413 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.450328    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78aa99a1-a611-4e29-98d0-5095769e04b8-openshift-service-ca\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.450413 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.450364    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbzf8\" (UniqueName: \"kubernetes.io/projected/78aa99a1-a611-4e29-98d0-5095769e04b8-kube-api-access-bbzf8\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.450620 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.450509    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78aa99a1-a611-4e29-98d0-5095769e04b8-maas-api-tls\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.450620 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.450552    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78aa99a1-a611-4e29-98d0-5095769e04b8-metrics-certs\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.451096 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.451070    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78aa99a1-a611-4e29-98d0-5095769e04b8-openshift-service-ca\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.453100 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.453073    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78aa99a1-a611-4e29-98d0-5095769e04b8-maas-api-tls\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.453202 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.453103    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/78aa99a1-a611-4e29-98d0-5095769e04b8-metrics-certs\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.457604 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.457582    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbzf8\" (UniqueName: \"kubernetes.io/projected/78aa99a1-a611-4e29-98d0-5095769e04b8-kube-api-access-bbzf8\") pod \"maas-api-65c67cbdfc-wk2rl\" (UID: \"78aa99a1-a611-4e29-98d0-5095769e04b8\") " pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.485538 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.485513    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:06.760487 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:06.760450    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"]
Sep 10 21:34:06.762280 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:34:06.762250    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78aa99a1_a611_4e29_98d0_5095769e04b8.slice/crio-f2f1de121c3d90ae03dc1dd62457c160c311afc5d71ee9524b8cf8ffc03c1def WatchSource:0}: Error finding container f2f1de121c3d90ae03dc1dd62457c160c311afc5d71ee9524b8cf8ffc03c1def: Status 404 returned error can't find the container with id f2f1de121c3d90ae03dc1dd62457c160c311afc5d71ee9524b8cf8ffc03c1def
Sep 10 21:34:07.252008 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:07.251971    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl" event={"ID":"78aa99a1-a611-4e29-98d0-5095769e04b8","Type":"ContainerStarted","Data":"f2f1de121c3d90ae03dc1dd62457c160c311afc5d71ee9524b8cf8ffc03c1def"}
Sep 10 21:34:08.259017 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:08.258986    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl" event={"ID":"78aa99a1-a611-4e29-98d0-5095769e04b8","Type":"ContainerStarted","Data":"ecac2cdba06e68bf0d1bcd984b6ed906b394cc9090817fa20f824081f47816f9"}
Sep 10 21:34:08.259437 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:08.259150    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:08.278984 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:08.278929    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl" podStartSLOduration=1.573365742 podStartE2EDuration="2.27891024s" totalImagesPullingTime="705.544498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:34:08.277109697 +0000 UTC m=+2135.279356043" watchObservedRunningTime="2026-09-10 21:34:08.27891024 +0000 UTC m=+2135.281156588"
Sep 10 21:34:14.269597 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:14.269568    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-65c67cbdfc-wk2rl"
Sep 10 21:34:14.317022 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:14.316989    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-nxd8f"]
Sep 10 21:34:14.317290 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:14.317247    2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f" podUID="02867d1d-96c4-493a-95f9-b9a3711b0808" containerName="maas-api" containerID="cri-o://48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f" gracePeriod=30
Sep 10 21:34:15.564590 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.564567    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:34:15.618939 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.618904    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqtk4\" (UniqueName: \"kubernetes.io/projected/02867d1d-96c4-493a-95f9-b9a3711b0808-kube-api-access-rqtk4\") pod \"02867d1d-96c4-493a-95f9-b9a3711b0808\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") "
Sep 10 21:34:15.619088 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.618957    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02867d1d-96c4-493a-95f9-b9a3711b0808-openshift-service-ca\") pod \"02867d1d-96c4-493a-95f9-b9a3711b0808\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") "
Sep 10 21:34:15.619088 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.618981    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-maas-api-tls\") pod \"02867d1d-96c4-493a-95f9-b9a3711b0808\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") "
Sep 10 21:34:15.619088 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.619055    2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-metrics-certs\") pod \"02867d1d-96c4-493a-95f9-b9a3711b0808\" (UID: \"02867d1d-96c4-493a-95f9-b9a3711b0808\") "
Sep 10 21:34:15.619383 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.619357    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02867d1d-96c4-493a-95f9-b9a3711b0808-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "02867d1d-96c4-493a-95f9-b9a3711b0808" (UID: "02867d1d-96c4-493a-95f9-b9a3711b0808"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 21:34:15.621100 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.621073    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02867d1d-96c4-493a-95f9-b9a3711b0808-kube-api-access-rqtk4" (OuterVolumeSpecName: "kube-api-access-rqtk4") pod "02867d1d-96c4-493a-95f9-b9a3711b0808" (UID: "02867d1d-96c4-493a-95f9-b9a3711b0808"). InnerVolumeSpecName "kube-api-access-rqtk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 21:34:15.621240 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.621204    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "02867d1d-96c4-493a-95f9-b9a3711b0808" (UID: "02867d1d-96c4-493a-95f9-b9a3711b0808"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:34:15.621299 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.621231    2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "02867d1d-96c4-493a-95f9-b9a3711b0808" (UID: "02867d1d-96c4-493a-95f9-b9a3711b0808"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 21:34:15.719662 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.719584    2580 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-metrics-certs\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:34:15.719662 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.719611    2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqtk4\" (UniqueName: \"kubernetes.io/projected/02867d1d-96c4-493a-95f9-b9a3711b0808-kube-api-access-rqtk4\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:34:15.719662 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.719622    2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/02867d1d-96c4-493a-95f9-b9a3711b0808-openshift-service-ca\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:34:15.719662 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:15.719631    2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/02867d1d-96c4-493a-95f9-b9a3711b0808-maas-api-tls\") on node \"ip-10-0-128-236.ec2.internal\" DevicePath \"\""
Sep 10 21:34:16.311083 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.311045    2580 generic.go:356] "Generic (PLEG): container finished" podID="02867d1d-96c4-493a-95f9-b9a3711b0808" containerID="48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f" exitCode=0
Sep 10 21:34:16.311255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.311093    2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f"
Sep 10 21:34:16.311255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.311158    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f" event={"ID":"02867d1d-96c4-493a-95f9-b9a3711b0808","Type":"ContainerDied","Data":"48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f"}
Sep 10 21:34:16.311255 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.311227    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-554988677c-nxd8f" event={"ID":"02867d1d-96c4-493a-95f9-b9a3711b0808","Type":"ContainerDied","Data":"9639896b1b5abca2f68d6cfd06b5795ddc6030848a758c8e7e2d0a060fe27f7b"}
Sep 10 21:34:16.311360 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.311260    2580 scope.go:117] "RemoveContainer" containerID="48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f"
Sep 10 21:34:16.320951 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.320935    2580 scope.go:117] "RemoveContainer" containerID="48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f"
Sep 10 21:34:16.321246 ip-10-0-128-236 kubenswrapper[2580]: E0910 21:34:16.321226    2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f\": container with ID starting with 48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f not found: ID does not exist" containerID="48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f"
Sep 10 21:34:16.321312 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.321255    2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f"} err="failed to get container status \"48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f\": rpc error: code = NotFound desc = could not find container \"48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f\": container with ID starting with 48a7b30a1c4f476453b2a5f20f654b234fe5a33294829b33eaa8be2f2823ef7f not found: ID does not exist"
Sep 10 21:34:16.332935 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.332912    2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-nxd8f"]
Sep 10 21:34:16.334978 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:16.334960    2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-554988677c-nxd8f"]
Sep 10 21:34:17.518803 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:17.518770    2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02867d1d-96c4-493a-95f9-b9a3711b0808" path="/var/lib/kubelet/pods/02867d1d-96c4-493a-95f9-b9a3711b0808/volumes"
Sep 10 21:34:36.357016 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:36.356984    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57c46d699b-thm79_f227f4ee-7aa6-4b25-a136-ef6e86804794/authorino/0.log"
Sep 10 21:34:42.175348 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:42.175318    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84497f567c-2m9rg_26d9dc4f-9497-47d9-9d3a-2db00d1d3882/manager/0.log"
Sep 10 21:34:42.575336 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:42.575261    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d67fdbbcb-j64nn_e2f792bf-bba1-4120-b9cb-7d4d9f64ab37/copy-manifests/0.log"
Sep 10 21:34:42.581664 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:42.581646    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d67fdbbcb-j64nn_e2f792bf-bba1-4120-b9cb-7d4d9f64ab37/manager/0.log"
Sep 10 21:34:43.308749 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:43.308718    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-65c67cbdfc-wk2rl_78aa99a1-a611-4e29-98d0-5095769e04b8/maas-api/0.log"
Sep 10 21:34:43.651067 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:43.651044    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-pbtt2_f2b80b23-09d4-42fb-91ad-d710c658485a/postgres/0.log"
Sep 10 21:34:44.475869 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:44.475839    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57c46d699b-thm79_f227f4ee-7aa6-4b25-a136-ef6e86804794/authorino/0.log"
Sep 10 21:34:44.710881 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:44.710853    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-g2wsv_a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77/manager/0.log"
Sep 10 21:34:44.961646 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:44.961616    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-mgjv5_69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f/manager/0.log"
Sep 10 21:34:45.212739 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:45.212651    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kxlwp_9fb7cd0c-d3c9-4532-9612-aaff905a9b0e/manager/0.log"
Sep 10 21:34:46.321901 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:46.321869    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mg8df_1aa96712-b017-464c-b322-0ef8cbd99c8f/payload-processing/0.log"
Sep 10 21:34:46.441562 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:46.441536    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bb47ccbbb-cpmj2_ee33f0ef-90c8-4e2c-b34d-0459c159036d/router/0.log"
Sep 10 21:34:46.926010 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:46.925985    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vswwq_013c0199-7a19-460d-8352-3d3cc3ed4062/main/0.log"
Sep 10 21:34:53.291409 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.291374    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n84km/must-gather-9bzmq"]
Sep 10 21:34:53.291931 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.291814    2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02867d1d-96c4-493a-95f9-b9a3711b0808" containerName="maas-api"
Sep 10 21:34:53.291931 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.291834    2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="02867d1d-96c4-493a-95f9-b9a3711b0808" containerName="maas-api"
Sep 10 21:34:53.292076 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.291946    2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="02867d1d-96c4-493a-95f9-b9a3711b0808" containerName="maas-api"
Sep 10 21:34:53.295257 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.295238    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n84km/must-gather-9bzmq"
Sep 10 21:34:53.297706 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.297684    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n84km\"/\"openshift-service-ca.crt\""
Sep 10 21:34:53.297807 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.297684    2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n84km\"/\"kube-root-ca.crt\""
Sep 10 21:34:53.297807 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.297685    2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n84km\"/\"default-dockercfg-xm668\""
Sep 10 21:34:53.303018 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.302973    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n84km/must-gather-9bzmq"]
Sep 10 21:34:53.419997 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.419951    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtwhx\" (UniqueName: \"kubernetes.io/projected/a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9-kube-api-access-dtwhx\") pod \"must-gather-9bzmq\" (UID: \"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9\") " pod="openshift-must-gather-n84km/must-gather-9bzmq"
Sep 10 21:34:53.419997 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.420008    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9-must-gather-output\") pod \"must-gather-9bzmq\" (UID: \"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9\") " pod="openshift-must-gather-n84km/must-gather-9bzmq"
Sep 10 21:34:53.521018 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.520989    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtwhx\" (UniqueName: \"kubernetes.io/projected/a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9-kube-api-access-dtwhx\") pod \"must-gather-9bzmq\" (UID: \"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9\") " pod="openshift-must-gather-n84km/must-gather-9bzmq"
Sep 10 21:34:53.521187 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.521027    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9-must-gather-output\") pod \"must-gather-9bzmq\" (UID: \"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9\") " pod="openshift-must-gather-n84km/must-gather-9bzmq"
Sep 10 21:34:53.521341 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.521325    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9-must-gather-output\") pod \"must-gather-9bzmq\" (UID: \"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9\") " pod="openshift-must-gather-n84km/must-gather-9bzmq"
Sep 10 21:34:53.532428 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.532402    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtwhx\" (UniqueName: \"kubernetes.io/projected/a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9-kube-api-access-dtwhx\") pod \"must-gather-9bzmq\" (UID: \"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9\") " pod="openshift-must-gather-n84km/must-gather-9bzmq"
Sep 10 21:34:53.605528 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.605462    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n84km/must-gather-9bzmq"
Sep 10 21:34:53.877227 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.877197    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n84km/must-gather-9bzmq"]
Sep 10 21:34:53.878573 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:34:53.878548    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda66d0b82_b8a3_4487_9fd1_6fd9dd4d56f9.slice/crio-672990b7e20c391a5b8aaba258c22b3bd2e522932c8d8d02bd4c945e2b096c4f WatchSource:0}: Error finding container 672990b7e20c391a5b8aaba258c22b3bd2e522932c8d8d02bd4c945e2b096c4f: Status 404 returned error can't find the container with id 672990b7e20c391a5b8aaba258c22b3bd2e522932c8d8d02bd4c945e2b096c4f
Sep 10 21:34:53.880481 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:53.880461    2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 21:34:54.514650 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:54.514621    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n84km/must-gather-9bzmq" event={"ID":"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9","Type":"ContainerStarted","Data":"672990b7e20c391a5b8aaba258c22b3bd2e522932c8d8d02bd4c945e2b096c4f"}
Sep 10 21:34:55.533344 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:55.533300    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n84km/must-gather-9bzmq" event={"ID":"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9","Type":"ContainerStarted","Data":"340a58f80d17841a985066736b91e83e2a76f6698300bb90de0ac506fe7f1430"}
Sep 10 21:34:55.533873 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:55.533352    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n84km/must-gather-9bzmq" event={"ID":"a66d0b82-b8a3-4487-9fd1-6fd9dd4d56f9","Type":"ContainerStarted","Data":"a8e83f0b565ff5703bdb1aa63d4e65f5e63143c7d2fa2f23f937ab9bfcc12b40"}
Sep 10 21:34:55.551582 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:55.551516    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n84km/must-gather-9bzmq" podStartSLOduration=1.759633148 podStartE2EDuration="2.55149856s" totalImagesPullingTime="791.865412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 21:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:34:55.548519142 +0000 UTC m=+2182.550765489" watchObservedRunningTime="2026-09-10 21:34:55.55149856 +0000 UTC m=+2182.553744908"
Sep 10 21:34:56.277944 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:56.277911    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ks95v_0cf2fb43-34ef-4996-b746-31482ad247ab/global-pull-secret-syncer/0.log"
Sep 10 21:34:56.385228 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:56.385197    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wq6lz_0b7e62b8-d8b2-431e-8d20-27577cd0f03c/konnectivity-agent/0.log"
Sep 10 21:34:56.407783 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:34:56.407752    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-236.ec2.internal_dc4b39e75aad3eda53248a12852cd31a/haproxy/0.log"
Sep 10 21:35:00.371018 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:00.370967    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57c46d699b-thm79_f227f4ee-7aa6-4b25-a136-ef6e86804794/authorino/0.log"
Sep 10 21:35:00.446144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:00.445373    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-g2wsv_a4bd7e78-6c5f-47bd-8ea7-d2696b0f1d77/manager/0.log"
Sep 10 21:35:00.515149 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:00.514211    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-mgjv5_69aaf1ba-f13d-4ac9-a703-ab278a7ffe6f/manager/0.log"
Sep 10 21:35:00.612144 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:00.611005    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-kxlwp_9fb7cd0c-d3c9-4532-9612-aaff905a9b0e/manager/0.log"
Sep 10 21:35:02.727586 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:02.727553    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-cnxrr_c33e975b-b71c-4b5c-af0a-936e5a809ae5/monitoring-plugin/0.log"
Sep 10 21:35:02.829783 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:02.829749    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwdzd_19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef/node-exporter/0.log"
Sep 10 21:35:02.850467 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:02.850354    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwdzd_19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef/kube-rbac-proxy/0.log"
Sep 10 21:35:02.869319 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:02.869286    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-gwdzd_19a6ea4d-d3fd-4c0e-a687-ee0bae8a60ef/init-textfile/0.log"
Sep 10 21:35:02.971267 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:02.971234    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-ks5dc_5492f628-1e3d-4399-9e78-e509ebd03d25/kube-rbac-proxy-main/0.log"
Sep 10 21:35:02.995026 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:02.994919    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-ks5dc_5492f628-1e3d-4399-9e78-e509ebd03d25/kube-rbac-proxy-self/0.log"
Sep 10 21:35:03.024608 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:03.024566    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-ks5dc_5492f628-1e3d-4399-9e78-e509ebd03d25/openshift-state-metrics/0.log"
Sep 10 21:35:03.236247 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:03.236214    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-49rkn_be2b2bed-d84a-440b-ace3-c746319c2dd0/prometheus-operator/0.log"
Sep 10 21:35:03.254564 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:03.254476    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-49rkn_be2b2bed-d84a-440b-ace3-c746319c2dd0/kube-rbac-proxy/0.log"
Sep 10 21:35:06.438228 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.438198    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-554c9578b8-mvb89_6a31ddc8-6c68-4dc5-9211-2eece856ccbc/console/0.log"
Sep 10 21:35:06.861655 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.861566    2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"]
Sep 10 21:35:06.864926 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.864894    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:06.875165 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.875139    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"]
Sep 10 21:35:06.980728 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.980692    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-sys\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:06.981032 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.981010    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-proc\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:06.981301 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.981279    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnzwj\" (UniqueName: \"kubernetes.io/projected/6b24a3c5-be75-4d21-8357-0a6cf99147cd-kube-api-access-bnzwj\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:06.981454 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.981437    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-lib-modules\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:06.981615 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:06.981599    2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-podres\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.082979 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.082937    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnzwj\" (UniqueName: \"kubernetes.io/projected/6b24a3c5-be75-4d21-8357-0a6cf99147cd-kube-api-access-bnzwj\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.083194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.082996    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-lib-modules\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.083194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.083037    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-podres\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.083194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.083086    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-sys\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.083194 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.083148    2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-proc\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.083449 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.083294    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-proc\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.083669 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.083642    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-sys\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.083869 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.083837    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-podres\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.083965 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.083888    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6b24a3c5-be75-4d21-8357-0a6cf99147cd-lib-modules\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.092179 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.092155    2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnzwj\" (UniqueName: \"kubernetes.io/projected/6b24a3c5-be75-4d21-8357-0a6cf99147cd-kube-api-access-bnzwj\") pod \"perf-node-gather-daemonset-x5njq\" (UID: \"6b24a3c5-be75-4d21-8357-0a6cf99147cd\") " pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.160563 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.160531    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-l726r_72eb50b8-401f-4e0a-8416-1901732a1e19/volume-data-source-validator/0.log"
Sep 10 21:35:07.178081 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.178048    2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.482111 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.481996    2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"]
Sep 10 21:35:07.487262 ip-10-0-128-236 kubenswrapper[2580]: W0910 21:35:07.487227    2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b24a3c5_be75_4d21_8357_0a6cf99147cd.slice/crio-c71a2c9c7b255c106ab197672a68830ca6093913a8e215c6230b3930f81d7454 WatchSource:0}: Error finding container c71a2c9c7b255c106ab197672a68830ca6093913a8e215c6230b3930f81d7454: Status 404 returned error can't find the container with id c71a2c9c7b255c106ab197672a68830ca6093913a8e215c6230b3930f81d7454
Sep 10 21:35:07.655671 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.654648    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq" event={"ID":"6b24a3c5-be75-4d21-8357-0a6cf99147cd","Type":"ContainerStarted","Data":"e8344bf20d962f248638e5e7fc9ccc484864e252541aaddd204ab978f943b8de"}
Sep 10 21:35:07.655671 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.654717    2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq" event={"ID":"6b24a3c5-be75-4d21-8357-0a6cf99147cd","Type":"ContainerStarted","Data":"c71a2c9c7b255c106ab197672a68830ca6093913a8e215c6230b3930f81d7454"}
Sep 10 21:35:07.655671 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.655604    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:07.684877 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:07.684802    2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq" podStartSLOduration=1.6847794280000001 podStartE2EDuration="1.684779428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 21:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 21:35:07.681636559 +0000 UTC m=+2194.683882929" watchObservedRunningTime="2026-09-10 21:35:07.684779428 +0000 UTC m=+2194.687025777"
Sep 10 21:35:08.455504 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:08.455477    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l68d2_107d30b3-f5f5-455b-8834-509a7b83772f/dns/0.log"
Sep 10 21:35:08.475606 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:08.475570    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-l68d2_107d30b3-f5f5-455b-8834-509a7b83772f/kube-rbac-proxy/0.log"
Sep 10 21:35:08.498439 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:08.498412    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gp8hn_7ff24c61-0ba5-4a58-a41c-95284c27b8d3/dns-node-resolver/0.log"
Sep 10 21:35:09.113320 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:09.113273    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctnsl_372eb974-c525-4f33-954a-23c699ac2016/node-ca/0.log"
Sep 10 21:35:10.468234 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:10.468205    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mg8df_1aa96712-b017-464c-b322-0ef8cbd99c8f/payload-processing/0.log"
Sep 10 21:35:10.491450 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:10.491417    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bb47ccbbb-cpmj2_ee33f0ef-90c8-4e2c-b34d-0459c159036d/router/0.log"
Sep 10 21:35:11.362342 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:11.362313    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fwkbp_2ba3f4ef-5e26-47d9-92b0-37bae0dd5650/serve-healthcheck-canary/0.log"
Sep 10 21:35:11.885744 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:11.885708    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dprcj_1d024674-eb1f-40e7-960c-d7d8ea66eb75/kube-rbac-proxy/0.log"
Sep 10 21:35:11.905727 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:11.905702    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dprcj_1d024674-eb1f-40e7-960c-d7d8ea66eb75/exporter/0.log"
Sep 10 21:35:11.925102 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:11.925076    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dprcj_1d024674-eb1f-40e7-960c-d7d8ea66eb75/extractor/0.log"
Sep 10 21:35:14.344113 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:14.344076    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-84497f567c-2m9rg_26d9dc4f-9497-47d9-9d3a-2db00d1d3882/manager/0.log"
Sep 10 21:35:14.467061 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:14.467018    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d67fdbbcb-j64nn_e2f792bf-bba1-4120-b9cb-7d4d9f64ab37/manager/0.log"
Sep 10 21:35:14.485278 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:14.485249    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-d67fdbbcb-j64nn_e2f792bf-bba1-4120-b9cb-7d4d9f64ab37/copy-manifests/0.log"
Sep 10 21:35:14.682446 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:14.682414    2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n84km/perf-node-gather-daemonset-x5njq"
Sep 10 21:35:16.001868 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:16.001838    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-548fd67646-q8cd6_e841aab1-748a-4125-a278-a9d9330fdd1c/manager/0.log"
Sep 10 21:35:16.857191 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:16.857081    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mg8df_1aa96712-b017-464c-b322-0ef8cbd99c8f/payload-processing/0.log"
Sep 10 21:35:16.870623 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:16.870594    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bb47ccbbb-cpmj2_ee33f0ef-90c8-4e2c-b34d-0459c159036d/router/0.log"
Sep 10 21:35:18.664566 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:18.664456    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-mg8df_1aa96712-b017-464c-b322-0ef8cbd99c8f/payload-processing/0.log"
Sep 10 21:35:18.678748 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:18.678209    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7bb47ccbbb-cpmj2_ee33f0ef-90c8-4e2c-b34d-0459c159036d/router/0.log"
Sep 10 21:35:21.564663 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:21.564627    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-4f6l7_26a53a6f-5cc0-4659-ad6d-cc250ea634af/kube-storage-version-migrator-operator/1.log"
Sep 10 21:35:21.565537 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:21.565508    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-4f6l7_26a53a6f-5cc0-4659-ad6d-cc250ea634af/kube-storage-version-migrator-operator/0.log"
Sep 10 21:35:22.834520 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:22.834492    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pcz7g_1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25/kube-multus-additional-cni-plugins/0.log"
Sep 10 21:35:22.856968 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:22.856939    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pcz7g_1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25/egress-router-binary-copy/0.log"
Sep 10 21:35:22.877307 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:22.877279    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pcz7g_1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25/cni-plugins/0.log"
Sep 10 21:35:22.895024 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:22.895003    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pcz7g_1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25/bond-cni-plugin/0.log"
Sep 10 21:35:22.913047 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:22.913026    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pcz7g_1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25/routeoverride-cni/0.log"
Sep 10 21:35:22.930846 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:22.930825    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pcz7g_1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25/whereabouts-cni-bincopy/0.log"
Sep 10 21:35:22.948699 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:22.948681    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-pcz7g_1fc6d8e4-36e1-4853-a3cf-b6cb1a26ed25/whereabouts-cni/0.log"
Sep 10 21:35:23.067292 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:23.067267    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xcgkb_4a77b4af-edcb-478c-8d57-4b7b29339a22/kube-multus/0.log"
Sep 10 21:35:23.090302 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:23.090218    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8wqmx_9341a00d-f1e0-4798-b687-8f058c4e2d59/network-metrics-daemon/0.log"
Sep 10 21:35:23.106473 ip-10-0-128-236 kubenswrapper[2580]: I0910 21:35:23.106450    2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8wqmx_9341a00d-f1e0-4798-b687-8f058c4e2d59/kube-rbac-proxy/0.log"