Sep 08 16:12:45.810015 ip-10-0-128-92 systemd[1]: Starting Kubernetes Kubelet...
Sep 08 16:12:46.171464 ip-10-0-128-92 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:12:46.171464 ip-10-0-128-92 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 08 16:12:46.171464 ip-10-0-128-92 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:12:46.171464 ip-10-0-128-92 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 08 16:12:46.171464 ip-10-0-128-92 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 08 16:12:46.172351 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.172263    2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 08 16:12:46.176150 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176133    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:46.176150 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176148    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:46.176150 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176153    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:46.176150 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176157    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176161    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176165    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176169    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176174    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176178    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176182    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176186    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176191    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176195    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176199    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176203    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176207    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176210    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176214    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176219    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176222    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176226    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176230    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:46.176388 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176234    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176238    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176242    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176246    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176250    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176254    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176261    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176267    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176272    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176278    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176282    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176286    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176290    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176295    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176299    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176304    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176309    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176313    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176317    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176321    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:46.177173 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176325    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176330    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176334    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176338    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176342    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176346    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176350    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176354    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176359    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176363    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176367    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176371    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176376    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176380    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176384    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176388    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176392    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176400    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176406    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176411    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:46.177736 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176416    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176420    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176424    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176429    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176433    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176437    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176441    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176447    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176454    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176460    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176465    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176469    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176474    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176479    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176483    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176487    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176491    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176496    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176503    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:46.178216 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176508    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176513    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176517    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176521    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176525    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176529    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176533    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176538    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176542    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176546    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176550    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176554    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176558    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176563    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176568    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176572    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176591    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.176596    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177625    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177636    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:46.178707 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177640    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177643    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177645    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177650    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177654    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177657    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177659    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177662    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177665    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177668    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177670    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177673    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177675    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177678    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177681    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177684    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177686    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177689    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177692    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:46.179183 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177694    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177697    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177699    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177702    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177705    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177707    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177710    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177712    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177715    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177717    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177720    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177723    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177728    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177731    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177733    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177736    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177739    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177741    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177744    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177746    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:46.179752 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177749    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177754    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177757    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177761    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177765    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177768    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177771    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177774    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177777    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177780    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177783    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177786    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177789    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177792    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177795    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177797    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177800    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177803    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177806    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:46.180289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177808    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177811    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177813    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177816    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177818    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177821    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177825    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177828    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177830    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177833    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177835    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177839    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177842    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177846    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177850    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177853    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177855    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177858    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177860    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:46.180769 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177863    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177866    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177868    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177871    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177874    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177876    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177879    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177881    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177884    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177886    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177889    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177891    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177894    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177897    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177899    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177901    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177904    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177907    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177910    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.177913    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:46.181225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178273    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178278    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178281    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178283    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178286    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178289    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178292    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178295    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178297    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178300    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178303    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178305    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178308    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178310    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178313    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178316    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178318    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178321    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178324    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178326    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:46.181732 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178329    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178332    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178334    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178337    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178340    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178342    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178345    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178347    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178350    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178353    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178355    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178358    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178361    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178364    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178366    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178369    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178372    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178374    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178377    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178379    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:46.182231 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178382    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178384    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178387    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178389    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178391    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178396    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178399    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178402    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178405    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178408    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178411    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178413    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178416    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178419    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178421    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178424    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178426    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178429    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178431    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:46.182729 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178434    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178437    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178439    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178442    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178445    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178447    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178450    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178453    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178455    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178458    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178461    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178463    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178466    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178469    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178473    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178476    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178479    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178481    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178484    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178487    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:46.183198 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178491    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178493    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178495    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178498    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178501    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178504    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178506    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178509    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178511    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178514    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178516    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178519    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178522    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178524    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178527    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178532    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178535    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178537    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178540    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.178544    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:46.183733 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178628    2575 flags.go:64] FLAG: --address="0.0.0.0"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178663    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178669    2575 flags.go:64] FLAG: --anonymous-auth="true"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178673    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178677    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178681    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178685    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178689    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178693    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178696    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178700    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178703    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178706    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178709    2575 flags.go:64] FLAG: --cgroup-root=""
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178713    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178716    2575 flags.go:64] FLAG: --client-ca-file=""
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178719    2575 flags.go:64] FLAG: --cloud-provider="external"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178722    2575 flags.go:64] FLAG: --cluster-dns="[]"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178725    2575 flags.go:64] FLAG: --cluster-domain=""
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178728    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178731    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178734    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178737    2575 flags.go:64] FLAG: --container-log-max-files="5"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178741    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 08 16:12:46.184228 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178745    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178748    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178751    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178755    2575 flags.go:64] FLAG: --contention-profiling="false"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178758    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178761    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178764    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178767    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178771    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178774    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178778    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178780    2575 flags.go:64] FLAG: --enable-load-reader="false"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178783    2575 flags.go:64] FLAG: --enable-server="true"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178786    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178790    2575 flags.go:64] FLAG: --event-burst="100"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178793    2575 flags.go:64] FLAG: --event-qps="50"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178796    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178801    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178804    2575 flags.go:64] FLAG: --eviction-hard=""
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178807    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178810    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178813    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178816    2575 flags.go:64] FLAG: --eviction-soft=""
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178819    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178822    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 08 16:12:46.184809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178825    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178827    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178830    2575 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178833    2575 flags.go:64] FLAG: --fail-swap-on="true"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178836    2575 flags.go:64] FLAG: --feature-gates=""
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178840    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178843    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178846    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178850    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178853    2575 flags.go:64] FLAG: --healthz-port="10248"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178856    2575 flags.go:64] FLAG: --help="false"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178859    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178862    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178865    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178868    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178871    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178875    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178878    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178880    2575 flags.go:64] FLAG: --image-service-endpoint=""
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178883    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178886    2575 flags.go:64] FLAG: --kube-api-burst="100"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178889    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178892    2575 flags.go:64] FLAG: --kube-api-qps="50"
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178895    2575 flags.go:64] FLAG: --kube-reserved=""
Sep 08 16:12:46.185438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178899    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178902    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178905    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178908    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178911    2575 flags.go:64] FLAG: --lock-file=""
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178913    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178916    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178919    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178924    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178927    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178930    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178933    2575 flags.go:64] FLAG: --logging-format="text"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178936    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178939    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178942    2575 flags.go:64] FLAG: --manifest-url=""
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178945    2575 flags.go:64] FLAG: --manifest-url-header=""
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178949    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178952    2575 flags.go:64] FLAG: --max-open-files="1000000"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178956    2575 flags.go:64] FLAG: --max-pods="110"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178959    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178962    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178964    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178968    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178971    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178974    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 08 16:12:46.186086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178977    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178984    2575 flags.go:64] FLAG: --node-status-max-images="50"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178987    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178990    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178993    2575 flags.go:64] FLAG: --pod-cidr=""
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.178996    2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179001    2575 flags.go:64] FLAG: --pod-manifest-path=""
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179006    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179009    2575 flags.go:64] FLAG: --pods-per-core="0"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179012    2575 flags.go:64] FLAG: --port="10250"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179015    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179017    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-051b3a1391d16e9b6"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179021    2575 flags.go:64] FLAG: --qos-reserved=""
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179023    2575 flags.go:64] FLAG: --read-only-port="10255"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179026    2575 flags.go:64] FLAG: --register-node="true"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179029    2575 flags.go:64] FLAG: --register-with-taints=""
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179032    2575 flags.go:64] FLAG: --registry-burst="10"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179035    2575 flags.go:64] FLAG: --registry-qps="5"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179038    2575 flags.go:64] FLAG: --reserved-cpus=""
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179041    2575 flags.go:64] FLAG: --reserved-memory=""
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179044    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179047    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179050    2575 flags.go:64] FLAG: --rotate-certificates="false"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179053    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179056    2575 flags.go:64] FLAG: --runonce="false"
Sep 08 16:12:46.187026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179059    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179062    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179064    2575 flags.go:64] FLAG: --seccomp-default="false"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179067    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179070    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179073    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179076    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179079    2575 flags.go:64] FLAG: --storage-driver-password="root"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179082    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179085    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179088    2575 flags.go:64] FLAG: --storage-driver-user="root"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179090    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179093    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179096    2575 flags.go:64] FLAG: --system-cgroups=""
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179101    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179106    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179109    2575 flags.go:64] FLAG: --tls-cert-file=""
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179111    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179114    2575 flags.go:64] FLAG: --tls-min-version=""
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179117    2575 flags.go:64] FLAG: --tls-private-key-file=""
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179120    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179123    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179125    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179128    2575 flags.go:64] FLAG: --v="2"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179132    2575 flags.go:64] FLAG: --version="false"
Sep 08 16:12:46.187704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179136    2575 flags.go:64] FLAG: --vmodule=""
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179140    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.179143    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179258    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179262    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179266    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179271    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179275    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179279    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179282    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179284    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179287    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179290    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179294    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179297    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179300    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179303    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179306    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:46.188340 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179308    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179311    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179313    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179317    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179320    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179322    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179324    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179327    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179330    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179332    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179335    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179338    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179341    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179343    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179346    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179348    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179351    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179356    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179358    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179361    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:46.188853 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179364    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179367    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179369    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179372    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179375    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179377    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179380    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179382    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179385    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179387    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179390    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179393    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179396    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179398    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179401    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179404    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179407    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179409    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179412    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:46.189347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179414    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179417    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179419    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179422    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179424    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179427    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179429    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179432    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179434    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179437    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179441    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179443    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179446    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179448    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179451    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179453    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179456    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179458    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179461    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179463    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:46.189870 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179466    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179469    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179471    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179474    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179476    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179479    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179482    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179484    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179488    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179490    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179493    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179496    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179498    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179501    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179503    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179506    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179508    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179511    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179513    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179516    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:46.190363 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179519    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179521    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179525    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179528    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.179530    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.180194    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.188308    2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.188325    2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.189118    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.189128    2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189185    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189190    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189193    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189196    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:46.190931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189199    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189203    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189206    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189208    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189211    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189214    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189217    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189219    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189222    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189225    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189227    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189230    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189233    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189235    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189238    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189241    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189244    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189247    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189249    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189252    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:46.191280 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189255    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189258    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189260    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189263    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189267    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189270    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189273    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189276    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189279    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189281    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189284    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189287    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189290    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189292    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189295    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189297    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189300    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189302    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189305    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189307    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:46.191786 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189310    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189312    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189316    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189319    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189322    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189325    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189328    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189330    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189333    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189335    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189338    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189340    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189343    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189346    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189348    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189351    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189354    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189358    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189361    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189363    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:46.192317 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189366    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189369    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189373    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189376    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189380    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189383    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189386    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189388    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189391    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189394    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189396    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189399    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189401    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189404    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189407    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189411    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189413    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189416    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189418    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189420    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:46.192834 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189423    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189425    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189429    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189432    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189435    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189438    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189441    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189443    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189446    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189449    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189452    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189454    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189457    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189460    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189463    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:46.193347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.189468    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189605    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189611    2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189615    2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189618    2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189621    2575 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189623    2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189627    2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189631    2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189634    2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189636    2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189639    2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189641    2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189645    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189647    2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189650    2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189653    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189656    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189658    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 08 16:12:46.193731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189661    2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189664    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189666    2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189669    2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189671    2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189673    2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189676    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189678    2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189681    2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189684    2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189687    2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189689    2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189692    2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189694    2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189697    2575 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189699    2575 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189702    2575 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189704    2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189707    2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189728    2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 08 16:12:46.194192 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189732    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189735    2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189738    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189741    2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189744    2575 feature_gate.go:328] unrecognized feature gate: Example2
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189747    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189750    2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189753    2575 feature_gate.go:328] unrecognized feature gate: Example
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189756    2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189759    2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189761    2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189764    2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189766    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189769    2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189771    2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189773    2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189776    2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189779    2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189781    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189784    2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 08 16:12:46.194697 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189787    2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189789    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189792    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189795    2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189797    2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189800    2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189802    2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189805    2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189808    2575 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189810    2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189813    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189815    2575 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189818    2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189820    2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189823    2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189826    2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189830    2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189833    2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189836    2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 08 16:12:46.195196 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189838    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189841    2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189844    2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189847    2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189849    2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189852    2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189854    2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189856    2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189859    2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189861    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189864    2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189866    2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189870    2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189873    2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189875    2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189879    2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189883    2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189886    2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189889    2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 08 16:12:46.195677 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189892    2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189895    2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:46.189897    2575 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.189902    2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.190012    2575 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.191871    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.192696    2575 server.go:1019] "Starting client certificate rotation"
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.192800    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.192835    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 08 16:12:46.196151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.195036    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 16:12:46.196425 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.196208    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 08 16:12:46.207851 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.207833    2575 log.go:25] "Validated CRI v1 runtime API"
Sep 08 16:12:46.210702 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.210689    2575 log.go:25] "Validated CRI v1 image API"
Sep 08 16:12:46.214222 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.214205    2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 08 16:12:46.217565 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.217541    2575 fs.go:135] Filesystem UUIDs: map[6be13846-8690-4e66-89d3-c90ab435499d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 be886bfc-9cc5-455e-bc7e-9e2ffd4be476:/dev/nvme0n1p4]
Sep 08 16:12:46.217565 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.217563    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 08 16:12:46.224773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.223269    2575 manager.go:216] Machine: {Timestamp:2026-09-08 16:12:46.222772422 +0000 UTC m=+0.309436160 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099914 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f0e6bb9da434371b60091f9956256 SystemUUID:ec2f0e6b-b9da-4343-71b6-0091f9956256 BootID:fe476cb5-dda5-426b-bc6f-15d4a54edc97 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bc:a3:17:cf:09 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bc:a3:17:cf:09 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:df:35:ab:16:55 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 08 16:12:46.224773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.224170    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 08 16:12:46.224773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.224358    2575 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 08 16:12:46.227953 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.227924    2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 08 16:12:46.228097 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.227958    2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-92.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 08 16:12:46.229055 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.229044    2575 topology_manager.go:138] "Creating topology manager with none policy"
Sep 08 16:12:46.229101 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.229057    2575 container_manager_linux.go:309] "Creating device plugin manager"
Sep 08 16:12:46.229101 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.229065    2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 16:12:46.229551 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.229541    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 08 16:12:46.230247 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.230236    2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 08 16:12:46.230446 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.230427    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 16:12:46.233549 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.233536    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 16:12:46.233680 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.233670    2575 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 08 16:12:46.233712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.233702    2575 kubelet.go:486] "Attempting to sync node with API server"
Sep 08 16:12:46.233712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.233712    2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 08 16:12:46.233764 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.233725    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 08 16:12:46.233764 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.233735    2575 kubelet.go:398] "Adding apiserver pod source"
Sep 08 16:12:46.233764 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.233744    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 08 16:12:46.234117 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.234105    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 16:12:46.234160 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.234126    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 08 16:12:46.236266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.236250    2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 08 16:12:46.238348 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.238334    2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 08 16:12:46.238394 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.238365    2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 08 16:12:46.239627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239611    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 08 16:12:46.239669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239632    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 08 16:12:46.239669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239638    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 08 16:12:46.239669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239644    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 08 16:12:46.239669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239652    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 08 16:12:46.239669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239657    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 08 16:12:46.239669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239670    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 08 16:12:46.239817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239676    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 08 16:12:46.239817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239683    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 08 16:12:46.239817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239688    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 08 16:12:46.239817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239697    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 08 16:12:46.239817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239702    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 08 16:12:46.239817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239729    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 08 16:12:46.239817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.239734    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 08 16:12:46.243257 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.243238    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-92.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 08 16:12:46.243369 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.243356    2575 server.go:1268] "Started kubelet"
Sep 08 16:12:46.243495 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.243448    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 08 16:12:46.243549 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.243519    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 08 16:12:46.243609 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.243542    2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 08 16:12:46.243905 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.243865    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 08 16:12:46.244011 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.243938    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 08 16:12:46.244205 ip-10-0-128-92 systemd[1]: Started Kubernetes Kubelet.
Sep 08 16:12:46.245349 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.245332    2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 08 16:12:46.246787 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.246266    2575 server.go:310] "Adding debug handlers to kubelet server"
Sep 08 16:12:46.249373 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.249352    2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 08 16:12:46.250483 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.250466    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 08 16:12:46.250738 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.249831    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-92.ec2.internal.18d3647d48534ec1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-92.ec2.internal,UID:ip-10-0-128-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-92.ec2.internal,},FirstTimestamp:2026-09-08 16:12:46.243327681 +0000 UTC m=+0.329991423,LastTimestamp:2026-09-08 16:12:46.243327681 +0000 UTC m=+0.329991423,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-92.ec2.internal,}"
Sep 08 16:12:46.251001 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.250980    2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 08 16:12:46.251165 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.251149    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:46.251562 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.251543    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 08 16:12:46.251562 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.251543    2575 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 08 16:12:46.251733 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.251588    2575 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 08 16:12:46.251733 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.251660    2575 reconstruct.go:97] "Volume reconstruction finished"
Sep 08 16:12:46.251733 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.251667    2575 reconciler.go:29] "Reconciler: start to sync state"
Sep 08 16:12:46.252231 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.252216    2575 factory.go:55] Registering systemd factory
Sep 08 16:12:46.252287 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.252240    2575 factory.go:223] Registration of the systemd container factory successfully
Sep 08 16:12:46.252458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.252446    2575 factory.go:153] Registering CRI-O factory
Sep 08 16:12:46.252458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.252458    2575 factory.go:223] Registration of the crio container factory successfully
Sep 08 16:12:46.252553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.252496    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 08 16:12:46.252553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.252511    2575 factory.go:103] Registering Raw factory
Sep 08 16:12:46.252553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.252525    2575 manager.go:1196] Started watching for new ooms in manager
Sep 08 16:12:46.252861 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.252852    2575 manager.go:349] Starting recovery of all containers
Sep 08 16:12:46.257176 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.257116    2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-92.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 08 16:12:46.257340 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.257267    2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 08 16:12:46.265832 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.265779    2575 manager.go:354] Recovery completed
Sep 08 16:12:46.268159 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.268136    2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 08 16:12:46.271321 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.271306    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:46.275124 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.275098    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:46.275177 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.275138    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:46.275177 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.275148    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:46.275715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.275700    2575 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 08 16:12:46.275715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.275711    2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 08 16:12:46.275830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.275727    2575 state_mem.go:36] "Initialized new in-memory state store"
Sep 08 16:12:46.275830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.275709    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:46.276553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.276538    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:46.276649 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.276564    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:46.276649 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.276592    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:46.276881 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.276870    2575 policy_none.go:49] "None policy: Start"
Sep 08 16:12:46.276928 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.276884    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 08 16:12:46.276928 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.276894    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 08 16:12:46.277175 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.277064    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-92.ec2.internal.18d3647d4a387b21  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-92.ec2.internal,UID:ip-10-0-128-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-92.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-92.ec2.internal,},FirstTimestamp:2026-09-08 16:12:46.275124001 +0000 UTC m=+0.361787739,LastTimestamp:2026-09-08 16:12:46.275124001 +0000 UTC m=+0.361787739,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-92.ec2.internal,}"
Sep 08 16:12:46.277869 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.277858    2575 policy_none.go:47] "Start"
Sep 08 16:12:46.280445 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.280424    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2dt5g"
Sep 08 16:12:46.286182 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.286109    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-92.ec2.internal.18d3647d4a38c3eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-92.ec2.internal,UID:ip-10-0-128-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-92.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-92.ec2.internal,},FirstTimestamp:2026-09-08 16:12:46.275142635 +0000 UTC m=+0.361806373,LastTimestamp:2026-09-08 16:12:46.275142635 +0000 UTC m=+0.361806373,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-92.ec2.internal,}"
Sep 08 16:12:46.294172 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.294154    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2dt5g"
Sep 08 16:12:46.295545 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.295462    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-92.ec2.internal.18d3647d4a38e7f9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-92.ec2.internal,UID:ip-10-0-128-92.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-92.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-92.ec2.internal,},FirstTimestamp:2026-09-08 16:12:46.275151865 +0000 UTC m=+0.361815605,LastTimestamp:2026-09-08 16:12:46.275151865 +0000 UTC m=+0.361815605,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-92.ec2.internal,}"
Sep 08 16:12:46.304158 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.304142    2575 manager.go:337] "Starting Device Plugin manager"
Sep 08 16:12:46.304249 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.304189    2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 08 16:12:46.304249 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.304202    2575 server.go:87] "Starting device plugin registration server"
Sep 08 16:12:46.304457 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.304443    2575 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 08 16:12:46.304519 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.304458    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 08 16:12:46.304564 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.304535    2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 08 16:12:46.304645 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.304630    2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 08 16:12:46.304697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.304647    2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 08 16:12:46.305251 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.305213    2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 08 16:12:46.305316 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.305285    2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:46.324765 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.324742    2575 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2607 into the system container \"/system.slice\": <nil>"
Sep 08 16:12:46.405351 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.405326    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:46.406467 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.406435    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:46.406570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.406483    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:46.406570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.406514    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:46.406570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.406545    2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.417706 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.417685    2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.429689 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.417711    2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-92.ec2.internal\": node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:46.438067 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.438048    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:46.498726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.498691    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 08 16:12:46.499913 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.499889    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 08 16:12:46.500001 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.499968    2575 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 08 16:12:46.500001 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.499997    2575 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 08 16:12:46.500100 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.500042    2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 08 16:12:46.502269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.502245    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:46.538335 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.538304    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:46.600706 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.600675    2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal"]
Sep 08 16:12:46.600767 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.600747    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:46.601730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.601716    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:46.601796 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.601743    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:46.601796 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.601752    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:46.604354 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.604343    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:46.604497 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.604472    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.604530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.604515    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:46.605105 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.605078    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:46.605105 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.605106    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:46.605239 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.605117    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:46.605239 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.605144    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:46.605239 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.605161    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:46.605239 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.605170    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:46.606142 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.606127    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.606232 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.606148    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:46.606798 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.606783    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:46.606888 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.606806    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:46.606888 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.606816    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:46.630161 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.630136    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-92.ec2.internal\" not found" node="ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.634415 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.634401    2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-92.ec2.internal\" not found" node="ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.638392 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.638378    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:46.652967 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.652953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15056cf40b6914fa82d7b40377d740f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal\" (UID: \"c15056cf40b6914fa82d7b40377d740f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.653020 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.652975    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15056cf40b6914fa82d7b40377d740f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal\" (UID: \"c15056cf40b6914fa82d7b40377d740f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.653020 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.652993    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1bd51e38f398bda4d9557423e80c6a1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.ec2.internal\" (UID: \"b1bd51e38f398bda4d9557423e80c6a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.739027 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.738983    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:46.753498 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.753479    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15056cf40b6914fa82d7b40377d740f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal\" (UID: \"c15056cf40b6914fa82d7b40377d740f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.753548 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.753504    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15056cf40b6914fa82d7b40377d740f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal\" (UID: \"c15056cf40b6914fa82d7b40377d740f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.753548 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.753521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1bd51e38f398bda4d9557423e80c6a1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.ec2.internal\" (UID: \"b1bd51e38f398bda4d9557423e80c6a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.753635 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.753548    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b1bd51e38f398bda4d9557423e80c6a1-config\") pod \"kube-apiserver-proxy-ip-10-0-128-92.ec2.internal\" (UID: \"b1bd51e38f398bda4d9557423e80c6a1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.753635 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.753570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c15056cf40b6914fa82d7b40377d740f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal\" (UID: \"c15056cf40b6914fa82d7b40377d740f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.753635 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.753595    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c15056cf40b6914fa82d7b40377d740f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal\" (UID: \"c15056cf40b6914fa82d7b40377d740f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.839666 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.839633    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:46.932127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.932103    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.937754 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:46.937736    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:46.939866 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:46.939758    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:47.040624 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:47.040603    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:47.141662 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:47.141633    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:47.194628 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.194565    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 08 16:12:47.195154 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.194810    2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 08 16:12:47.242190 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:47.242149    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:47.299311 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.299071    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-07 16:07:47 +0000 UTC" deadline="2028-05-20 10:46:28.775155334 +0000 UTC"
Sep 08 16:12:47.299311 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.299261    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14874h33m41.475898032s"
Sep 08 16:12:47.344833 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:47.344811    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:47.352638 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.352623    2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 08 16:12:47.355105 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.355090    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientMemory"
Sep 08 16:12:47.355160 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.355121    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasNoDiskPressure"
Sep 08 16:12:47.355160 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.355137    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeHasSufficientPID"
Sep 08 16:12:47.445169 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:47.445138    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:47.545344 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:47.545311    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:47.572543 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.572483    2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:47.613163 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:47.613115    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc15056cf40b6914fa82d7b40377d740f.slice/crio-4f8c40160e38de1882ba8891ea7207fd0e0ad9db8c74ad434f029f62c3efb945 WatchSource:0}: Error finding container 4f8c40160e38de1882ba8891ea7207fd0e0ad9db8c74ad434f029f62c3efb945: Status 404 returned error can't find the container with id 4f8c40160e38de1882ba8891ea7207fd0e0ad9db8c74ad434f029f62c3efb945
Sep 08 16:12:47.618528 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.618509    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:12:47.646410 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:47.646378    2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-92.ec2.internal\" not found"
Sep 08 16:12:47.682119 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.682096    2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:47.731828 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.731804    2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:47.744014 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:47.743988    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bd51e38f398bda4d9557423e80c6a1.slice/crio-ec3ba9c64068672663aa603d3e7ecf16dfd78265e3f52d280d52010a0fd42834 WatchSource:0}: Error finding container ec3ba9c64068672663aa603d3e7ecf16dfd78265e3f52d280d52010a0fd42834: Status 404 returned error can't find the container with id ec3ba9c64068672663aa603d3e7ecf16dfd78265e3f52d280d52010a0fd42834
Sep 08 16:12:47.751914 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.751894    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:47.764076 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.764059    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 16:12:47.765230 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.765217    2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal"
Sep 08 16:12:47.773902 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:47.773888    2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 08 16:12:48.234886 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.234847    2575 apiserver.go:52] "Watching apiserver"
Sep 08 16:12:48.243362 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.243335    2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 08 16:12:48.243848 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.243823    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bbgqp","kube-system/konnectivity-agent-vd2fl","kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp","openshift-cluster-node-tuning-operator/tuned-7v9th","openshift-image-registry/node-ca-dxwz6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal","openshift-multus/multus-l2lgl","openshift-network-operator/iptables-alerter-l982x","kube-system/global-pull-secret-syncer-fdnml","openshift-dns/node-resolver-9l8gd","openshift-multus/multus-additional-cni-plugins-lp26m","openshift-multus/network-metrics-daemon-d7gtt","openshift-network-diagnostics/network-check-target-vl6bn"]
Sep 08 16:12:48.245524 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.245489    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.245655 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.245585    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:12:48.246810 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.246792    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:48.246885 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.246857    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:12:48.248256 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.248234    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:48.248346 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.248307    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:12:48.249326 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.249306    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.250873 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.250850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.251029 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.251014    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 08 16:12:48.252084 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.252062    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-l4g28\""
Sep 08 16:12:48.252323 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.252304    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 08 16:12:48.252459 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.252438    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 08 16:12:48.252673 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.252647    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.252756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.252674    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 08 16:12:48.253299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.253177    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:12:48.253299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.253218    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 08 16:12:48.253299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.253260    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-95v2w\""
Sep 08 16:12:48.253545 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.253525    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.254871 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.254831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.254989 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.254938    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 08 16:12:48.254989 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.254949    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b88bx\""
Sep 08 16:12:48.255295 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.255270    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 08 16:12:48.255408 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.255364    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 08 16:12:48.256684 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.256445    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:12:48.257613 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.257490    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 08 16:12:48.258439 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.258424    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 08 16:12:48.258687 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.258647    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 08 16:12:48.259940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.259833    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 08 16:12:48.261295 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.260294    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x7mxl\""
Sep 08 16:12:48.261295 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.260480    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.261295 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.260547    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 08 16:12:48.261295 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.260648    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 08 16:12:48.261295 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.260484    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:12:48.261616 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261383    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 08 16:12:48.261616 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261539    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.261726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysctl-d\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.261726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68cba019-22fb-4c75-a02b-07238e18a14f-etc-tuned\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.261830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261762    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b74qn\" (UniqueName: \"kubernetes.io/projected/0d770632-49ab-40c5-87f5-4420e74a1a44-kube-api-access-b74qn\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:48.261830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261795    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68cba019-22fb-4c75-a02b-07238e18a14f-tmp\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.261923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.261923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261860    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-systemd\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.261923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-host\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.262074 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261928    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lwpz\" (UniqueName: \"kubernetes.io/projected/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-kube-api-access-4lwpz\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.262074 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.261987    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.262074 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262028    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxt57\" (UniqueName: \"kubernetes.io/projected/2b9dec8a-be9a-4102-9a20-4767b175e9cd-kube-api-access-bxt57\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.262535 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262345    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4lwn4\""
Sep 08 16:12:48.262535 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262373    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nznrf\""
Sep 08 16:12:48.262535 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262396    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 08 16:12:48.262748 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-run\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.262748 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-sys\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.262909 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262758    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-var-lib-kubelet\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.262909 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262786    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:48.262909 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262816    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysconfig\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.262909 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v68j\" (UniqueName: \"kubernetes.io/projected/68cba019-22fb-4c75-a02b-07238e18a14f-kube-api-access-9v68j\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.262909 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262902    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-host\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.263159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262932    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4133bb28-b1e2-4b78-9dc3-40043ee78b17-dbus\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.263159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.262977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-registration-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.263159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263078    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-device-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.263159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263142    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4133bb28-b1e2-4b78-9dc3-40043ee78b17-kubelet-config\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.263356 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263194    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.263356 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263246    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-socket-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.263356 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-sys-fs\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.263356 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263341    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:48.263529 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263385    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-modprobe-d\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.263529 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysctl-conf\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.263615 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263567    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.263615 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263572    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-serviceca\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.263725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263651    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-kubernetes\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.263725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.263690    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-lib-modules\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.265098 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265071    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 08 16:12:48.265193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265124    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 08 16:12:48.265193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265170    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 08 16:12:48.265533 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265516    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 08 16:12:48.265607 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265535    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 08 16:12:48.265678 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265656    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 08 16:12:48.265741 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265691    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x9vp6\""
Sep 08 16:12:48.265799 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265753    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 08 16:12:48.265925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.265908    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-6ksrb\""
Sep 08 16:12:48.266028 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.266011    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 08 16:12:48.266159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.266139    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 08 16:12:48.266250 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.266166    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zt44w\""
Sep 08 16:12:48.266250 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.266177    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 08 16:12:48.272411 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.272118    2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 08 16:12:48.291588 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.291564    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xw4zp"
Sep 08 16:12:48.300979 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.300956    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xw4zp"
Sep 08 16:12:48.353105 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.353078    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 08 16:12:48.364265 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364240    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-k8s-cni-cncf-io\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.364368 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/989fb239-f40a-405e-a570-8477f5c2a739-agent-certs\") pod \"konnectivity-agent-vd2fl\" (UID: \"989fb239-f40a-405e-a570-8477f5c2a739\") " pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:12:48.364368 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364301    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqwfp\" (UniqueName: \"kubernetes.io/projected/0cab0104-2fcc-4735-a30b-b39095652998-kube-api-access-zqwfp\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.364368 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364323    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:48.364368 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-serviceca\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.364546 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364372    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef6c43f1-b4e3-464f-a4ab-57c86138262b-hosts-file\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.364546 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364428    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-kubernetes\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.364677 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-lib-modules\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.364677 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364610    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-kubernetes\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.364677 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-slash\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.364677 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364652    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-host-slash\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.364677 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364668    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-hostroot\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-cni-bin\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364724    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-kubelet\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364744    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-lib-modules\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364753    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-ovn\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-systemd\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-host\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364914    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-systemd\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-serviceca\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364958    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-host\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364979    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lwpz\" (UniqueName: \"kubernetes.io/projected/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-kube-api-access-4lwpz\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.364996    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365016    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-cni-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-conf-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365114    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-etc-selinux\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365126    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365151    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-var-lib-kubelet\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365170    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-cni-bin\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovnkube-script-lib\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365249    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-var-lib-kubelet\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.365260    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:48.365299 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365255    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwb9\" (UniqueName: \"kubernetes.io/projected/ef6c43f1-b4e3-464f-a4ab-57c86138262b-kube-api-access-rmwb9\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.365337    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:48.865305371 +0000 UTC m=+2.951969103 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365360    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-cnibin\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365382    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-cni-binary-copy\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365410    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysconfig\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4133bb28-b1e2-4b78-9dc3-40043ee78b17-dbus\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-registration-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365497    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-etc-kubernetes\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysconfig\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365516    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-systemd\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365544    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-registration-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365607    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365633    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-sys-fs\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-netns\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.365923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365679    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4133bb28-b1e2-4b78-9dc3-40043ee78b17-dbus\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlbp8\" (UniqueName: \"kubernetes.io/projected/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-kube-api-access-xlbp8\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-sys-fs\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-modprobe-d\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365764    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysctl-conf\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365778    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-modprobe-d\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365785    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-node-log\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365811    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-iptables-alerter-script\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365841    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-system-cni-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365863    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-cni-multus\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365878    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-daemon-config\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365897    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysctl-conf\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365923    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-systemd-units\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365949    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysctl-d\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365965    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68cba019-22fb-4c75-a02b-07238e18a14f-etc-tuned\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.365981    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b74qn\" (UniqueName: \"kubernetes.io/projected/0d770632-49ab-40c5-87f5-4420e74a1a44-kube-api-access-b74qn\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:48.366695 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366004    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-socket-dir-parent\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366045    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-etc-sysctl-d\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-etc-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366075    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-log-socket\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366099    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366124    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovn-node-metrics-cert\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68cba019-22fb-4c75-a02b-07238e18a14f-tmp\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366183    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366220    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-cni-netd\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366242    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rgws\" (UniqueName: \"kubernetes.io/projected/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-kube-api-access-5rgws\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-system-cni-dir\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366318    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxt57\" (UniqueName: \"kubernetes.io/projected/2b9dec8a-be9a-4102-9a20-4767b175e9cd-kube-api-access-bxt57\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366322    2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366362    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-cnibin\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.366321    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366394    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-multus-certs\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.366427    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret podName:4133bb28-b1e2-4b78-9dc3-40043ee78b17 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:48.866411672 +0000 UTC m=+2.953075400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret") pod "global-pull-secret-syncer-fdnml" (UID: "4133bb28-b1e2-4b78-9dc3-40043ee78b17") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:48.367416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovnkube-config\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6c43f1-b4e3-464f-a4ab-57c86138262b-tmp-dir\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366519    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-run\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366573    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-sys\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366620    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmnjq\" (UniqueName: \"kubernetes.io/projected/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-kube-api-access-hmnjq\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366646    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-kubelet\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366671    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-run-netns\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366648    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-run\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366694    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-env-overrides\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366712    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68cba019-22fb-4c75-a02b-07238e18a14f-sys\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366722    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v68j\" (UniqueName: \"kubernetes.io/projected/68cba019-22fb-4c75-a02b-07238e18a14f-kube-api-access-9v68j\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366761    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-host\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366789    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-device-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366816    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/989fb239-f40a-405e-a570-8477f5c2a739-konnectivity-ca\") pod \"konnectivity-agent-vd2fl\" (UID: \"989fb239-f40a-405e-a570-8477f5c2a739\") " pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-var-lib-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366863    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-host\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.368286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366879    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-device-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.369175 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4133bb28-b1e2-4b78-9dc3-40043ee78b17-kubelet-config\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.369175 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366934    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4133bb28-b1e2-4b78-9dc3-40043ee78b17-kubelet-config\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.369175 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-socket-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.369175 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.366989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-cni-binary-copy\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.369175 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.367061    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b9dec8a-be9a-4102-9a20-4767b175e9cd-socket-dir\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.369857 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.369840    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/68cba019-22fb-4c75-a02b-07238e18a14f-etc-tuned\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.369857 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.369853    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68cba019-22fb-4c75-a02b-07238e18a14f-tmp\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.379548 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.379529    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:48.379548 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.379547    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:48.379665 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.379557    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4hv77 for pod openshift-network-diagnostics/network-check-target-vl6bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:48.379665 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.379598    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77 podName:77c33fb0-b890-4124-a87d-2acf5ef7ce83 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:48.87958604 +0000 UTC m=+2.966249799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-4hv77" (UniqueName: "kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77") pod "network-check-target-vl6bn" (UID: "77c33fb0-b890-4124-a87d-2acf5ef7ce83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:48.381976 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.381958    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v68j\" (UniqueName: \"kubernetes.io/projected/68cba019-22fb-4c75-a02b-07238e18a14f-kube-api-access-9v68j\") pod \"tuned-7v9th\" (UID: \"68cba019-22fb-4c75-a02b-07238e18a14f\") " pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.382371 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.382349    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b74qn\" (UniqueName: \"kubernetes.io/projected/0d770632-49ab-40c5-87f5-4420e74a1a44-kube-api-access-b74qn\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:48.382438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.382424    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxt57\" (UniqueName: \"kubernetes.io/projected/2b9dec8a-be9a-4102-9a20-4767b175e9cd-kube-api-access-bxt57\") pod \"aws-ebs-csi-driver-node-bt9gp\" (UID: \"2b9dec8a-be9a-4102-9a20-4767b175e9cd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.383192 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.383168    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lwpz\" (UniqueName: \"kubernetes.io/projected/6f8613b9-7b07-468f-a48c-ebb3d4d519e4-kube-api-access-4lwpz\") pod \"node-ca-dxwz6\" (UID: \"6f8613b9-7b07-468f-a48c-ebb3d4d519e4\") " pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.467967 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.467926    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-cni-binary-copy\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.467973    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-k8s-cni-cncf-io\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.467999    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/989fb239-f40a-405e-a570-8477f5c2a739-agent-certs\") pod \"konnectivity-agent-vd2fl\" (UID: \"989fb239-f40a-405e-a570-8477f5c2a739\") " pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:12:48.468113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468025    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqwfp\" (UniqueName: \"kubernetes.io/projected/0cab0104-2fcc-4735-a30b-b39095652998-kube-api-access-zqwfp\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.468113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468064    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef6c43f1-b4e3-464f-a4ab-57c86138262b-hosts-file\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.468113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-slash\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468113    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-host-slash\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.468339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-hostroot\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-cni-bin\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-kubelet\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468249    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-ovn\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-cni-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468328    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-conf-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-cni-bin\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468427    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovnkube-script-lib\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwb9\" (UniqueName: \"kubernetes.io/projected/ef6c43f1-b4e3-464f-a4ab-57c86138262b-kube-api-access-rmwb9\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468478    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-cnibin\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468504    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-cni-binary-copy\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468534    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-etc-kubernetes\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468551    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-cni-binary-copy\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468560    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-systemd\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468584    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468637    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-netns\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468675    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-k8s-cni-cncf-io\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468678    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlbp8\" (UniqueName: \"kubernetes.io/projected/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-kube-api-access-xlbp8\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468724    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-node-log\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468751    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-iptables-alerter-script\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-cni-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-system-cni-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468803    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-conf-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468807    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-cni-multus\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468833    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-daemon-config\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468843    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468858    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-systemd-units\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468880    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-cni-bin\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.468885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468889    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-socket-dir-parent\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468915    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-etc-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468941    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-log-socket\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468968    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468993    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovn-node-metrics-cert\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469037    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-cni-netd\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469065    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rgws\" (UniqueName: \"kubernetes.io/projected/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-kube-api-access-5rgws\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-system-cni-dir\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469121    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-cnibin\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-multus-certs\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469172    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovnkube-config\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6c43f1-b4e3-464f-a4ab-57c86138262b-tmp-dir\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469223    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469258    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmnjq\" (UniqueName: \"kubernetes.io/projected/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-kube-api-access-hmnjq\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-kubelet\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-run-netns\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469337    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-env-overrides\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469370    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/989fb239-f40a-405e-a570-8477f5c2a739-konnectivity-ca\") pod \"konnectivity-agent-vd2fl\" (UID: \"989fb239-f40a-405e-a570-8477f5c2a739\") " pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:12:48.469948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469397    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-var-lib-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovnkube-script-lib\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469500    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-var-lib-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469541    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-cnibin\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.469948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469900    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-netns\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.469948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469917    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-etc-kubernetes\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.469948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469929    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-systemd\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.469948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469936    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.468621    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-ovn\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.469994    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-etc-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470042    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-log-socket\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470062    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-cni-binary-copy\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470090    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470129    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-cni-multus\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470194    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-node-log\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470198    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-cnibin\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470261    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-cni-netd\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470267    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-var-lib-kubelet\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470290    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-cni-bin\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470329    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-hostroot\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.470358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470261    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-slash\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470388    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-run-openvswitch\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470378    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-system-cni-dir\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470434    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-systemd-units\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470455    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-socket-dir-parent\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470490    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-host-slash\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470497    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-host-run-multus-certs\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470538    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-kubelet\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470582    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-host-run-netns\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470668    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-multus-daemon-config\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470735    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cab0104-2fcc-4735-a30b-b39095652998-system-cni-dir\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470864    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-iptables-alerter-script\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470929    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ef6c43f1-b4e3-464f-a4ab-57c86138262b-hosts-file\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470965    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-env-overrides\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.470974    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovnkube-config\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.471038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.471026    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6c43f1-b4e3-464f-a4ab-57c86138262b-tmp-dir\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.471875 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.471069    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cab0104-2fcc-4735-a30b-b39095652998-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.472064 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.472038    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/989fb239-f40a-405e-a570-8477f5c2a739-konnectivity-ca\") pod \"konnectivity-agent-vd2fl\" (UID: \"989fb239-f40a-405e-a570-8477f5c2a739\") " pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:12:48.473171 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.473148    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-ovn-node-metrics-cert\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.474787 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.474765    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/989fb239-f40a-405e-a570-8477f5c2a739-agent-certs\") pod \"konnectivity-agent-vd2fl\" (UID: \"989fb239-f40a-405e-a570-8477f5c2a739\") " pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:12:48.478610 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.478570    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlbp8\" (UniqueName: \"kubernetes.io/projected/70fb5cab-3d58-4c00-8a8c-e02ec76cda46-kube-api-access-xlbp8\") pod \"multus-l2lgl\" (UID: \"70fb5cab-3d58-4c00-8a8c-e02ec76cda46\") " pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.480253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.480206    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqwfp\" (UniqueName: \"kubernetes.io/projected/0cab0104-2fcc-4735-a30b-b39095652998-kube-api-access-zqwfp\") pod \"multus-additional-cni-plugins-lp26m\" (UID: \"0cab0104-2fcc-4735-a30b-b39095652998\") " pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.481003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.480976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmnjq\" (UniqueName: \"kubernetes.io/projected/ab911ea9-7eae-4bcb-b27a-7bc33e6465a6-kube-api-access-hmnjq\") pod \"iptables-alerter-l982x\" (UID: \"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6\") " pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.481682 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.481618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rgws\" (UniqueName: \"kubernetes.io/projected/f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d-kube-api-access-5rgws\") pod \"ovnkube-node-bbgqp\" (UID: \"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.482745 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.482713    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwb9\" (UniqueName: \"kubernetes.io/projected/ef6c43f1-b4e3-464f-a4ab-57c86138262b-kube-api-access-rmwb9\") pod \"node-resolver-9l8gd\" (UID: \"ef6c43f1-b4e3-464f-a4ab-57c86138262b\") " pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.508291 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.508208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal" event={"ID":"b1bd51e38f398bda4d9557423e80c6a1","Type":"ContainerStarted","Data":"ec3ba9c64068672663aa603d3e7ecf16dfd78265e3f52d280d52010a0fd42834"}
Sep 08 16:12:48.510315 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.510284    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal" event={"ID":"c15056cf40b6914fa82d7b40377d740f","Type":"ContainerStarted","Data":"4f8c40160e38de1882ba8891ea7207fd0e0ad9db8c74ad434f029f62c3efb945"}
Sep 08 16:12:48.566280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.566250    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp"
Sep 08 16:12:48.572926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.572884    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7v9th"
Sep 08 16:12:48.580773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.580727    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dxwz6"
Sep 08 16:12:48.588116 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.587948    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l982x"
Sep 08 16:12:48.595890 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.595870    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-l2lgl"
Sep 08 16:12:48.599426 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:48.599401    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68cba019_22fb_4c75_a02b_07238e18a14f.slice/crio-07d33cbad300c5361627b325c3dac132c8311cfc88a570a279dc26290b8b5b71 WatchSource:0}: Error finding container 07d33cbad300c5361627b325c3dac132c8311cfc88a570a279dc26290b8b5b71: Status 404 returned error can't find the container with id 07d33cbad300c5361627b325c3dac132c8311cfc88a570a279dc26290b8b5b71
Sep 08 16:12:48.600066 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:48.599990    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8613b9_7b07_468f_a48c_ebb3d4d519e4.slice/crio-3031f65f6cd2e2c1b48bcce79aefd7bb2a52cb342d685c794723d691e6d24639 WatchSource:0}: Error finding container 3031f65f6cd2e2c1b48bcce79aefd7bb2a52cb342d685c794723d691e6d24639: Status 404 returned error can't find the container with id 3031f65f6cd2e2c1b48bcce79aefd7bb2a52cb342d685c794723d691e6d24639
Sep 08 16:12:48.601717 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.601696    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:12:48.603565 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:48.603539    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab911ea9_7eae_4bcb_b27a_7bc33e6465a6.slice/crio-72b1b1a47aafb4948948ecdf042d351212cab1cc27636c3582cc909dddbbaf1e WatchSource:0}: Error finding container 72b1b1a47aafb4948948ecdf042d351212cab1cc27636c3582cc909dddbbaf1e: Status 404 returned error can't find the container with id 72b1b1a47aafb4948948ecdf042d351212cab1cc27636c3582cc909dddbbaf1e
Sep 08 16:12:48.608148 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.608090    2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 08 16:12:48.609822 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.609693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:12:48.614050 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:48.614028    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70fb5cab_3d58_4c00_8a8c_e02ec76cda46.slice/crio-a664340680987b145ae03277ea216624c112443c2906dbf1df899562bfeec331 WatchSource:0}: Error finding container a664340680987b145ae03277ea216624c112443c2906dbf1df899562bfeec331: Status 404 returned error can't find the container with id a664340680987b145ae03277ea216624c112443c2906dbf1df899562bfeec331
Sep 08 16:12:48.617916 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.617906    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9l8gd"
Sep 08 16:12:48.624743 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.624723    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lp26m"
Sep 08 16:12:48.626470 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:48.626447    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod989fb239_f40a_405e_a570_8477f5c2a739.slice/crio-e132c7c051a050ba0f463034cc5270e108cab6502105baa1764c911cdf4f7e77 WatchSource:0}: Error finding container e132c7c051a050ba0f463034cc5270e108cab6502105baa1764c911cdf4f7e77: Status 404 returned error can't find the container with id e132c7c051a050ba0f463034cc5270e108cab6502105baa1764c911cdf4f7e77
Sep 08 16:12:48.627293 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:48.627170    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf317bd8d_0f4d_4df4_a6d1_fed8d9b54e4d.slice/crio-f19026b3b769329a8ed7902d961e92200c74df711d42c6aa9a7979676df80684 WatchSource:0}: Error finding container f19026b3b769329a8ed7902d961e92200c74df711d42c6aa9a7979676df80684: Status 404 returned error can't find the container with id f19026b3b769329a8ed7902d961e92200c74df711d42c6aa9a7979676df80684
Sep 08 16:12:48.642146 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:48.642109    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6c43f1_b4e3_464f_a4ab_57c86138262b.slice/crio-cd156014863df24d50e95f6d25e7550420cba5a1e2accab355958d303ab51819 WatchSource:0}: Error finding container cd156014863df24d50e95f6d25e7550420cba5a1e2accab355958d303ab51819: Status 404 returned error can't find the container with id cd156014863df24d50e95f6d25e7550420cba5a1e2accab355958d303ab51819
Sep 08 16:12:48.642801 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:12:48.642776    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cab0104_2fcc_4735_a30b_b39095652998.slice/crio-8eab3330319be9f77f7e599bcd291719940943339cd4f95e9de214416917190e WatchSource:0}: Error finding container 8eab3330319be9f77f7e599bcd291719940943339cd4f95e9de214416917190e: Status 404 returned error can't find the container with id 8eab3330319be9f77f7e599bcd291719940943339cd4f95e9de214416917190e
Sep 08 16:12:48.872527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.872435    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:48.872527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.872515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:48.872761 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.872676    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:48.872761 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.872684    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:48.872761 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.872735    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret podName:4133bb28-b1e2-4b78-9dc3-40043ee78b17 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:49.872720886 +0000 UTC m=+3.959384615 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret") pod "global-pull-secret-syncer-fdnml" (UID: "4133bb28-b1e2-4b78-9dc3-40043ee78b17") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:48.872761 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.872751    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:49.872743868 +0000 UTC m=+3.959407595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:48.972978 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:48.972944    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:48.973126 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.973100    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:48.973126 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.973124    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:48.973231 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.973134    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4hv77 for pod openshift-network-diagnostics/network-check-target-vl6bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:48.973231 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:48.973193    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77 podName:77c33fb0-b890-4124-a87d-2acf5ef7ce83 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:49.973173868 +0000 UTC m=+4.059837630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hv77" (UniqueName: "kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77") pod "network-check-target-vl6bn" (UID: "77c33fb0-b890-4124-a87d-2acf5ef7ce83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:49.303908 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.303785    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:07:49 +0000 UTC" deadline="2028-05-17 04:54:46.784264534 +0000 UTC"
Sep 08 16:12:49.303908 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.303847    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14796h41m57.480421212s"
Sep 08 16:12:49.502013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.501028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:49.502013 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.501149    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:12:49.514545 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.514505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vd2fl" event={"ID":"989fb239-f40a-405e-a570-8477f5c2a739","Type":"ContainerStarted","Data":"e132c7c051a050ba0f463034cc5270e108cab6502105baa1764c911cdf4f7e77"}
Sep 08 16:12:49.516510 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.516479    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l2lgl" event={"ID":"70fb5cab-3d58-4c00-8a8c-e02ec76cda46","Type":"ContainerStarted","Data":"a664340680987b145ae03277ea216624c112443c2906dbf1df899562bfeec331"}
Sep 08 16:12:49.519108 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.519075    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l982x" event={"ID":"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6","Type":"ContainerStarted","Data":"72b1b1a47aafb4948948ecdf042d351212cab1cc27636c3582cc909dddbbaf1e"}
Sep 08 16:12:49.525600 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.525571    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7v9th" event={"ID":"68cba019-22fb-4c75-a02b-07238e18a14f","Type":"ContainerStarted","Data":"07d33cbad300c5361627b325c3dac132c8311cfc88a570a279dc26290b8b5b71"}
Sep 08 16:12:49.530816 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.530789    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp" event={"ID":"2b9dec8a-be9a-4102-9a20-4767b175e9cd","Type":"ContainerStarted","Data":"e69ab32ac6dcf3c9d86c98bc5d28a70feeba699891a24d132d7292964286d032"}
Sep 08 16:12:49.534257 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.534229    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9l8gd" event={"ID":"ef6c43f1-b4e3-464f-a4ab-57c86138262b","Type":"ContainerStarted","Data":"cd156014863df24d50e95f6d25e7550420cba5a1e2accab355958d303ab51819"}
Sep 08 16:12:49.540369 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.540342    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"f19026b3b769329a8ed7902d961e92200c74df711d42c6aa9a7979676df80684"}
Sep 08 16:12:49.543170 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.543145    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxwz6" event={"ID":"6f8613b9-7b07-468f-a48c-ebb3d4d519e4","Type":"ContainerStarted","Data":"3031f65f6cd2e2c1b48bcce79aefd7bb2a52cb342d685c794723d691e6d24639"}
Sep 08 16:12:49.546499 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.546472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerStarted","Data":"8eab3330319be9f77f7e599bcd291719940943339cd4f95e9de214416917190e"}
Sep 08 16:12:49.880055 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.879994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:49.880258 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.880090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:49.880258 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.880249    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:49.880374 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.880323    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:51.880301524 +0000 UTC m=+5.966965255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:49.880776 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.880754    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:49.880877 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.880815    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret podName:4133bb28-b1e2-4b78-9dc3-40043ee78b17 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:51.880796296 +0000 UTC m=+5.967460025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret") pod "global-pull-secret-syncer-fdnml" (UID: "4133bb28-b1e2-4b78-9dc3-40043ee78b17") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:49.981640 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:49.980958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:49.981640 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.981142    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:49.981640 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.981166    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:49.981640 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.981180    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4hv77 for pod openshift-network-diagnostics/network-check-target-vl6bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:49.981640 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:49.981243    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77 podName:77c33fb0-b890-4124-a87d-2acf5ef7ce83 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:51.981222225 +0000 UTC m=+6.067885969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hv77" (UniqueName: "kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77") pod "network-check-target-vl6bn" (UID: "77c33fb0-b890-4124-a87d-2acf5ef7ce83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:50.304918 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:50.304817    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-07 16:07:49 +0000 UTC" deadline="2028-06-04 04:57:47.193971624 +0000 UTC"
Sep 08 16:12:50.304918 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:50.304864    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15228h44m56.88911312s"
Sep 08 16:12:50.503093 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:50.503060    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:50.503270 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:50.503241    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:12:50.503727 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:50.503706    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:50.503870 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:50.503848    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:12:51.500441 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:51.500406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:51.501065 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.500561    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:12:51.899465 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:51.898686    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:51.899465 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:51.898778    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:51.899465 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.898919    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:51.899465 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.898981    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:55.898961352 +0000 UTC m=+9.985625086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:51.899465 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.899379    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:51.899465 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.899428    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret podName:4133bb28-b1e2-4b78-9dc3-40043ee78b17 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:55.899412336 +0000 UTC m=+9.986076069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret") pod "global-pull-secret-syncer-fdnml" (UID: "4133bb28-b1e2-4b78-9dc3-40043ee78b17") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:51.999218 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:51.999179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:51.999391 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.999347    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:51.999391 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.999370    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:51.999391 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.999383    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4hv77 for pod openshift-network-diagnostics/network-check-target-vl6bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:51.999560 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:51.999440    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77 podName:77c33fb0-b890-4124-a87d-2acf5ef7ce83 nodeName:}" failed. No retries permitted until 2026-09-08 16:12:55.99942096 +0000 UTC m=+10.086084706 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hv77" (UniqueName: "kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77") pod "network-check-target-vl6bn" (UID: "77c33fb0-b890-4124-a87d-2acf5ef7ce83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:52.500575 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:52.500539    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:52.500575 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:52.500593    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:52.501126 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:52.500704    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:12:52.501126 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:52.500801    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:12:53.501242 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:53.500719    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:53.501242 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:53.500860    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:12:54.500509 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:54.500432    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:54.500702 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:54.500583    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:12:54.501039 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:54.501014    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:54.501158 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:54.501135    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:12:55.500563 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:55.500411    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:55.501212 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:55.500564    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:12:55.934516 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:55.933861    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:55.934516 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:55.933943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:55.934516 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:55.934135    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:55.934516 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:55.934206    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret podName:4133bb28-b1e2-4b78-9dc3-40043ee78b17 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:03.93418754 +0000 UTC m=+18.020851272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret") pod "global-pull-secret-syncer-fdnml" (UID: "4133bb28-b1e2-4b78-9dc3-40043ee78b17") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:12:55.934516 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:55.934287    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:55.934516 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:55.934328    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:03.934313884 +0000 UTC m=+18.020977615 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:12:56.035045 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:56.035001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:56.035239 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:56.035175    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:12:56.035239 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:56.035202    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:12:56.035239 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:56.035218    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4hv77 for pod openshift-network-diagnostics/network-check-target-vl6bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:56.035418 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:56.035291    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77 podName:77c33fb0-b890-4124-a87d-2acf5ef7ce83 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:04.035269936 +0000 UTC m=+18.121933680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hv77" (UniqueName: "kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77") pod "network-check-target-vl6bn" (UID: "77c33fb0-b890-4124-a87d-2acf5ef7ce83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:12:56.502884 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:56.502283    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:56.502884 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:56.502467    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:12:56.502884 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:56.502699    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:56.502884 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:56.502852    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:12:57.500242 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:57.500155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:57.500422 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:57.500289    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:12:58.500730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:58.500598    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:12:58.501339 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:58.501297    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:12:58.501463 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:58.501367    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:12:58.501529 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:58.501508    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:12:59.500307 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:12:59.500270    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:12:59.500477 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:12:59.500402    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:00.501159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:00.501121    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:00.501574 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:00.501257    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:00.501667 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:00.501129    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:00.501744 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:00.501722    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:01.500595 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:01.500543    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:01.500759 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:01.500683    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:02.500727 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:02.500693    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:02.501137 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:02.500700    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:02.501137 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:02.500830    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:02.501137 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:02.500910    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:03.500896 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:03.500863    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:03.501344 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:03.500977    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:04.001609 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:04.001475    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:04.001609 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:04.001567    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:04.001844 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.001667    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:04.001844 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.001735    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:04.001844 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.001736    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret podName:4133bb28-b1e2-4b78-9dc3-40043ee78b17 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:20.001722536 +0000 UTC m=+34.088386267 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret") pod "global-pull-secret-syncer-fdnml" (UID: "4133bb28-b1e2-4b78-9dc3-40043ee78b17") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:04.001844 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.001779    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:20.00176506 +0000 UTC m=+34.088428802 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:04.103019 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:04.102989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:04.103182 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.103149    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:13:04.103182 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.103171    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:13:04.103182 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.103181    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4hv77 for pod openshift-network-diagnostics/network-check-target-vl6bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:04.103294 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.103226    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77 podName:77c33fb0-b890-4124-a87d-2acf5ef7ce83 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:20.103213547 +0000 UTC m=+34.189877286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hv77" (UniqueName: "kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77") pod "network-check-target-vl6bn" (UID: "77c33fb0-b890-4124-a87d-2acf5ef7ce83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:04.500399 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:04.500310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:04.500548 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.500425    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:04.500548 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:04.500487    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:04.500654 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:04.500603    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:05.501109 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:05.501080    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:05.501553 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:05.501208    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:06.501526 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:06.501495    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:06.501968 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:06.501618    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:06.501968 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:06.501692    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:06.501968 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:06.501826    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:07.501783 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.501028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:07.501783 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:07.501147    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:07.589195 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.589010    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:13:07.589763 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.589733    2575 generic.go:356] "Generic (PLEG): container finished" podID="f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d" containerID="fc0ab21c63cab52abf700f199b9c78fb36066bd8ba66cf060a0300843ea4f84d" exitCode=1
Sep 08 16:13:07.589860 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.589810    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerDied","Data":"fc0ab21c63cab52abf700f199b9c78fb36066bd8ba66cf060a0300843ea4f84d"}
Sep 08 16:13:07.589914 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.589877    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"4a47eb56269bd09b57286535fa720379a021bb8755834e2bbf7f1fc72c437b77"}
Sep 08 16:13:07.592845 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.592817    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal" event={"ID":"b1bd51e38f398bda4d9557423e80c6a1","Type":"ContainerStarted","Data":"2450191499a344cb90a4712acc5e4d0fed3cadaf396edc531e2e8e4096177206"}
Sep 08 16:13:07.595836 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.595808    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-l2lgl" event={"ID":"70fb5cab-3d58-4c00-8a8c-e02ec76cda46","Type":"ContainerStarted","Data":"4cbb30ecffeb8f07285bb74d7ee96b8f7f7260ef185cf8585b9ec09b2a057b77"}
Sep 08 16:13:07.608492 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.608169    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-92.ec2.internal" podStartSLOduration=19.608150684 podStartE2EDuration="19.608150684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:07.608090683 +0000 UTC m=+21.694754431" watchObservedRunningTime="2026-09-08 16:13:07.608150684 +0000 UTC m=+21.694814433"
Sep 08 16:13:07.612867 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.612844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7v9th" event={"ID":"68cba019-22fb-4c75-a02b-07238e18a14f","Type":"ContainerStarted","Data":"ba43d05ff2cb7d9f98e3bfeeb221f357f2f76439cb1412ccf249d9cc3089c3e5"}
Sep 08 16:13:07.626279 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.626208    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-l2lgl" podStartSLOduration=1.884456568 podStartE2EDuration="20.626192522s" totalImagesPullingTime="18.741735954s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:07.624622837 +0000 UTC m=+21.711286587" watchObservedRunningTime="2026-09-08 16:13:07.626192522 +0000 UTC m=+21.712856270"
Sep 08 16:13:07.645787 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:07.645695    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7v9th" podStartSLOduration=1.941033596 podStartE2EDuration="20.645675804s" totalImagesPullingTime="18.704642208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:07.644799783 +0000 UTC m=+21.731463532" watchObservedRunningTime="2026-09-08 16:13:07.645675804 +0000 UTC m=+21.732339549"
Sep 08 16:13:08.500651 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.500621    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:08.500651 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.500637    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:08.500890 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:08.500731    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:08.500890 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:08.500860    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:08.619383 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.619298    2575 generic.go:356] "Generic (PLEG): container finished" podID="0cab0104-2fcc-4735-a30b-b39095652998" containerID="e385c979d8926a6637b5bb86f2ea7ea286562f4611e6926522fd72402a1baec8" exitCode=0
Sep 08 16:13:08.620224 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.619373    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerDied","Data":"e385c979d8926a6637b5bb86f2ea7ea286562f4611e6926522fd72402a1baec8"}
Sep 08 16:13:08.621638 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.621612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vd2fl" event={"ID":"989fb239-f40a-405e-a570-8477f5c2a739","Type":"ContainerStarted","Data":"6908cc78f36fbbe4b36be84b3e3e315d53bfac5f5b744aa12df8a2df07d0708f"}
Sep 08 16:13:08.625523 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.625472    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp" event={"ID":"2b9dec8a-be9a-4102-9a20-4767b175e9cd","Type":"ContainerStarted","Data":"04a29d33f1689ef670cda37252bbe552bb7e64f477f2bebfa77b19220f9feed9"}
Sep 08 16:13:08.627677 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.627649    2575 generic.go:356] "Generic (PLEG): container finished" podID="c15056cf40b6914fa82d7b40377d740f" containerID="c00252684f0faa0eff3eec9fca6473327a502be4b9569b1c7adb18cfe469db28" exitCode=0
Sep 08 16:13:08.627771 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.627717    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal" event={"ID":"c15056cf40b6914fa82d7b40377d740f","Type":"ContainerDied","Data":"c00252684f0faa0eff3eec9fca6473327a502be4b9569b1c7adb18cfe469db28"}
Sep 08 16:13:08.629363 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.629338    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9l8gd" event={"ID":"ef6c43f1-b4e3-464f-a4ab-57c86138262b","Type":"ContainerStarted","Data":"31c5bb28d89faa497b4b4a2718d93c828506d105f8dc14b117717492285182f2"}
Sep 08 16:13:08.632647 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.632625    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:13:08.633086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.633063    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"bbfb1a9d8b35f9b60b36cd37fc966dc33c023cddf1671faff5d6da79a1316348"}
Sep 08 16:13:08.633161 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.633123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"7cf25a23e0f68342197a82a1545975b603ff149ec96da30a8d0e912eb5e342de"}
Sep 08 16:13:08.633215 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.633161    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"a52db09d851244933b58c055600aa8f8196ea6d2a0579541be500ed2134e7e20"}
Sep 08 16:13:08.633215 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.633194    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"4c343b4235b79297b5ebc3afb9fbf28c66cb25d56401d5714c0335d13061e98a"}
Sep 08 16:13:08.634657 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.634634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dxwz6" event={"ID":"6f8613b9-7b07-468f-a48c-ebb3d4d519e4","Type":"ContainerStarted","Data":"9b2111c937d2b2a3b0ac085e9a62e5967c9da5698aa7fc1d86019410f977227a"}
Sep 08 16:13:08.656833 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.656789    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dxwz6" podStartSLOduration=2.993958898 podStartE2EDuration="21.656773747s" totalImagesPullingTime="18.662814849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:08.656652047 +0000 UTC m=+22.743315794" watchObservedRunningTime="2026-09-08 16:13:08.656773747 +0000 UTC m=+22.743437497"
Sep 08 16:13:08.670151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.670109    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vd2fl" podStartSLOduration=3.034959355 podStartE2EDuration="21.670096089s" totalImagesPullingTime="18.635136734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:08.670021103 +0000 UTC m=+22.756684850" watchObservedRunningTime="2026-09-08 16:13:08.670096089 +0000 UTC m=+22.756759845"
Sep 08 16:13:08.687473 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.687436    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9l8gd" podStartSLOduration=3.040620325 podStartE2EDuration="21.687424739s" totalImagesPullingTime="18.646804414s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:08.687138538 +0000 UTC m=+22.773802286" watchObservedRunningTime="2026-09-08 16:13:08.687424739 +0000 UTC m=+22.774088487"
Sep 08 16:13:08.848394 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:08.848247    2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 08 16:13:09.319570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.319476    2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-08T16:13:08.84839143Z","UUID":"7c7df35e-041d-4412-8023-9381ca770586","Handler":null,"Name":"","Endpoint":""}
Sep 08 16:13:09.321019 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.320993    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 08 16:13:09.321019 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.321020    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 08 16:13:09.500679 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.500648    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:09.500833 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:09.500740    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:09.639003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.638924    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l982x" event={"ID":"ab911ea9-7eae-4bcb-b27a-7bc33e6465a6","Type":"ContainerStarted","Data":"fb01b29a09fa4c62de1e392b47b8cb14dc140804cbae78ecbce4a879b72de7d8"}
Sep 08 16:13:09.641420 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.641307    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp" event={"ID":"2b9dec8a-be9a-4102-9a20-4767b175e9cd","Type":"ContainerStarted","Data":"e4b972ee4b3feaef470db64277b5d2636fb95e963100f7c27cf8d6d1781b3332"}
Sep 08 16:13:09.643638 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.643559    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal" event={"ID":"c15056cf40b6914fa82d7b40377d740f","Type":"ContainerStarted","Data":"0138c694fd4928790024bbc2920a9ac3030adc4ccf42a27a47725e188485df6e"}
Sep 08 16:13:09.653673 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.653635    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l982x" podStartSLOduration=3.946931609 podStartE2EDuration="22.653618401s" totalImagesPullingTime="18.706686792s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:09.653189552 +0000 UTC m=+23.739853301" watchObservedRunningTime="2026-09-08 16:13:09.653618401 +0000 UTC m=+23.740282152"
Sep 08 16:13:09.667542 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:09.667501    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-92.ec2.internal" podStartSLOduration=21.667488099 podStartE2EDuration="21.667488099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:09.667365164 +0000 UTC m=+23.754028914" watchObservedRunningTime="2026-09-08 16:13:09.667488099 +0000 UTC m=+23.754151846"
Sep 08 16:13:10.501180 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:10.501095    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:10.501338 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:10.501235    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:10.501338 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:10.501305    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:10.501451 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:10.501429    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:10.647805 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:10.647773    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp" event={"ID":"2b9dec8a-be9a-4102-9a20-4767b175e9cd","Type":"ContainerStarted","Data":"4c075d36f3fce19ce29ccb25db5576c4c9b06c53785d720d79463826021e8318"}
Sep 08 16:13:10.653259 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:10.653236    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:13:10.653723 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:10.653699    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"d57c12edd47dc2a57a9593987340a224c4b9b4b29aa4d6bf5041beec56818d58"}
Sep 08 16:13:11.500326 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:11.500299    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:11.500506 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:11.500419    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:12.500896 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:12.500814    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:12.501286 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:12.500932    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:12.501286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:12.500998    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:12.501286 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:12.501134    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:13.406900 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.406874    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:13:13.407381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.407364    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:13:13.420846 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.420811    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bt9gp" podStartSLOduration=5.508073723 podStartE2EDuration="26.420796907s" totalImagesPullingTime="20.912723184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:10.668062892 +0000 UTC m=+24.754726640" watchObservedRunningTime="2026-09-08 16:13:13.420796907 +0000 UTC m=+27.507460655"
Sep 08 16:13:13.500651 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.500626    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:13.500742 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:13.500727    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:13.663204 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.663053    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:13:13.663560 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.663537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"32fdf1ef1d3b63bd4f3e001cba4331b48713e8e6b21996b53e215feb5fa89812"}
Sep 08 16:13:13.666773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.664114    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:13:13.666773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.665730    2575 scope.go:117] "RemoveContainer" containerID="fc0ab21c63cab52abf700f199b9c78fb36066bd8ba66cf060a0300843ea4f84d"
Sep 08 16:13:13.666773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.665893    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:13:13.667120 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.667098    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:13:13.667251 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.667167    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:13:13.667251 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.667237    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vd2fl"
Sep 08 16:13:13.682268 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.682247    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:13:13.682754 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:13.682717    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:13:14.501253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:14.501230    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:14.501395 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:14.501318    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:14.501395 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:14.501380    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:14.501485 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:14.501468    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:14.668292 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:14.668260    2575 generic.go:356] "Generic (PLEG): container finished" podID="0cab0104-2fcc-4735-a30b-b39095652998" containerID="2bbfbc1b76006bd669f59cc01a1ed07b376d254115a20bb9a671ae18119e192d" exitCode=0
Sep 08 16:13:14.668876 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:14.668339    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerDied","Data":"2bbfbc1b76006bd669f59cc01a1ed07b376d254115a20bb9a671ae18119e192d"}
Sep 08 16:13:14.672932 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:14.672913    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:13:14.673378 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:14.673350    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" event={"ID":"f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d","Type":"ContainerStarted","Data":"d77b08d0fd470cf0e1a453cb78fc5188c8ca1f6f45c4baa6a40b8ec1a323eee1"}
Sep 08 16:13:14.715939 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:14.715889    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp" podStartSLOduration=9.042013874 podStartE2EDuration="27.715873212s" totalImagesPullingTime="18.673859338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:14.715757256 +0000 UTC m=+28.802421006" watchObservedRunningTime="2026-09-08 16:13:14.715873212 +0000 UTC m=+28.802536960"
Sep 08 16:13:15.446797 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:15.446765    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fdnml"]
Sep 08 16:13:15.446963 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:15.446933    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:15.447068 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:15.447039    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:15.449768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:15.449736    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d7gtt"]
Sep 08 16:13:15.449877 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:15.449846    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:15.449949 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:15.449927    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:15.450627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:15.450604    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vl6bn"]
Sep 08 16:13:15.450733 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:15.450705    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:15.450827 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:15.450792    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:15.678272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:15.678244    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerStarted","Data":"a396dd9bf3c99384e4c06025cd68f52c2837fc0c2676093a2cd2a705dd05b2ae"}
Sep 08 16:13:16.683023 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:16.682862    2575 generic.go:356] "Generic (PLEG): container finished" podID="0cab0104-2fcc-4735-a30b-b39095652998" containerID="a396dd9bf3c99384e4c06025cd68f52c2837fc0c2676093a2cd2a705dd05b2ae" exitCode=0
Sep 08 16:13:16.683023 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:16.682942    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerDied","Data":"a396dd9bf3c99384e4c06025cd68f52c2837fc0c2676093a2cd2a705dd05b2ae"}
Sep 08 16:13:17.500198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:17.500169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:17.500300 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:17.500169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:17.500300 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:17.500273    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:17.500408 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:17.500365    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:17.500408 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:17.500169    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:17.500506 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:17.500471    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:17.687001 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:17.686977    2575 generic.go:356] "Generic (PLEG): container finished" podID="0cab0104-2fcc-4735-a30b-b39095652998" containerID="3fdcc5b59d8de57172d7aa81fc8e480e3b064969c1a90bd50814fce70f260d8b" exitCode=0
Sep 08 16:13:17.687314 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:17.687037    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerDied","Data":"3fdcc5b59d8de57172d7aa81fc8e480e3b064969c1a90bd50814fce70f260d8b"}
Sep 08 16:13:19.501063 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:19.501028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:19.501520 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:19.501040    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:19.501520 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:19.501123    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vl6bn" podUID="77c33fb0-b890-4124-a87d-2acf5ef7ce83"
Sep 08 16:13:19.501520 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:19.501219    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fdnml" podUID="4133bb28-b1e2-4b78-9dc3-40043ee78b17"
Sep 08 16:13:19.501520 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:19.501043    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:19.501520 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:19.501388    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:13:20.027551 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.027507    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:20.027708 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.027623    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:20.027708 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.027672    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:20.027789 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.027711    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:20.027789 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.027743    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret podName:4133bb28-b1e2-4b78-9dc3-40043ee78b17 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:52.027726826 +0000 UTC m=+66.114390559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret") pod "global-pull-secret-syncer-fdnml" (UID: "4133bb28-b1e2-4b78-9dc3-40043ee78b17") : object "kube-system"/"original-pull-secret" not registered
Sep 08 16:13:20.027789 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.027760    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:52.027753416 +0000 UTC m=+66.114417141 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 08 16:13:20.128112 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.128027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:20.128250 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.128188    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 08 16:13:20.128250 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.128213    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 08 16:13:20.128250 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.128227    2575 projected.go:196] Error preparing data for projected volume kube-api-access-4hv77 for pod openshift-network-diagnostics/network-check-target-vl6bn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:20.128366 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.128289    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77 podName:77c33fb0-b890-4124-a87d-2acf5ef7ce83 nodeName:}" failed. No retries permitted until 2026-09-08 16:13:52.128270816 +0000 UTC m=+66.214934558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-4hv77" (UniqueName: "kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77") pod "network-check-target-vl6bn" (UID: "77c33fb0-b890-4124-a87d-2acf5ef7ce83") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 08 16:13:20.149453 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.149433    2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-92.ec2.internal" event="NodeReady"
Sep 08 16:13:20.149596 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.149543    2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 08 16:13:20.182563 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.182543    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c595d89b8-mxfw5"]
Sep 08 16:13:20.202723 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.202698    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sl68l"]
Sep 08 16:13:20.202853 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.202835    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.205351 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.205267    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 08 16:13:20.205351 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.205286    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 08 16:13:20.205543 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.205383    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 08 16:13:20.205543 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.205426    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgmfx\""
Sep 08 16:13:20.211042 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.211006    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 08 16:13:20.223001 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.222982    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c595d89b8-mxfw5"]
Sep 08 16:13:20.223105 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.223018    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-fnfbq"]
Sep 08 16:13:20.223176 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.223113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.225440 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.225422    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 08 16:13:20.225440 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.225428    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 08 16:13:20.225620 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.225485    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlhzx\""
Sep 08 16:13:20.246386 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.246364    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fnfbq"]
Sep 08 16:13:20.246493 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.246399    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sl68l"]
Sep 08 16:13:20.246493 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.246482    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fnfbq"
Sep 08 16:13:20.250024 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.249319    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 08 16:13:20.250024 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.249607    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h7rjg\""
Sep 08 16:13:20.250024 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.249866    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 08 16:13:20.250254 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.250218    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 08 16:13:20.329424 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvss\" (UniqueName: \"kubernetes.io/projected/d4599713-5ceb-4e41-a84e-0256209180b7-kube-api-access-8zvss\") pod \"ingress-canary-fnfbq\" (UID: \"d4599713-5ceb-4e41-a84e-0256209180b7\") " pod="openshift-ingress-canary/ingress-canary-fnfbq"
Sep 08 16:13:20.329556 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329435    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62641f8d-0486-444e-b887-a345668b437f-ca-trust-extracted\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.329556 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329461    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-installation-pull-secrets\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.329556 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329480    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-bound-sa-token\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.329556 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329536    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-image-registry-private-configuration\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.329756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfrnc\" (UniqueName: \"kubernetes.io/projected/ed691713-c786-4c14-b382-8bdedeb4073d-kube-api-access-dfrnc\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.329756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329617    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4599713-5ceb-4e41-a84e-0256209180b7-cert\") pod \"ingress-canary-fnfbq\" (UID: \"d4599713-5ceb-4e41-a84e-0256209180b7\") " pod="openshift-ingress-canary/ingress-canary-fnfbq"
Sep 08 16:13:20.329756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed691713-c786-4c14-b382-8bdedeb4073d-config-volume\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.329756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-trusted-ca\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.329756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6bxv\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-kube-api-access-z6bxv\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.329986 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329773    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-registry-certificates\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.329986 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329827    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed691713-c786-4c14-b382-8bdedeb4073d-tmp-dir\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.329986 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329864    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.329986 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.329892    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.430772 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.430692    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-image-registry-private-configuration\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.430772 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.430746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfrnc\" (UniqueName: \"kubernetes.io/projected/ed691713-c786-4c14-b382-8bdedeb4073d-kube-api-access-dfrnc\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.430772 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.430767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4599713-5ceb-4e41-a84e-0256209180b7-cert\") pod \"ingress-canary-fnfbq\" (UID: \"d4599713-5ceb-4e41-a84e-0256209180b7\") " pod="openshift-ingress-canary/ingress-canary-fnfbq"
Sep 08 16:13:20.431009 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.430798    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed691713-c786-4c14-b382-8bdedeb4073d-config-volume\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.431009 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.430817    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-trusted-ca\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.431009 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.430845    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6bxv\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-kube-api-access-z6bxv\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.431139 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431092    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-registry-certificates\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.431184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431136    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed691713-c786-4c14-b382-8bdedeb4073d-tmp-dir\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.431184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431166    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.431261 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.431261 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431239    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvss\" (UniqueName: \"kubernetes.io/projected/d4599713-5ceb-4e41-a84e-0256209180b7-kube-api-access-8zvss\") pod \"ingress-canary-fnfbq\" (UID: \"d4599713-5ceb-4e41-a84e-0256209180b7\") " pod="openshift-ingress-canary/ingress-canary-fnfbq"
Sep 08 16:13:20.431261 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431257    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62641f8d-0486-444e-b887-a345668b437f-ca-trust-extracted\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.431383 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431273    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-installation-pull-secrets\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.431383 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-bound-sa-token\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.431472 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431398    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ed691713-c786-4c14-b382-8bdedeb4073d-config-volume\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.431906 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.431600    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:20.431906 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.431676    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:13:20.931656153 +0000 UTC m=+35.018319891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:20.431906 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.431770    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:13:20.431906 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.431787    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:13:20.431906 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.431845    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:13:20.931827122 +0000 UTC m=+35.018490868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:13:20.431906 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431883    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62641f8d-0486-444e-b887-a345668b437f-ca-trust-extracted\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.431906 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431889    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed691713-c786-4c14-b382-8bdedeb4073d-tmp-dir\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.432306 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.431933    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-trusted-ca\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.432306 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.432259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-registry-certificates\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.435515 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.435494    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d4599713-5ceb-4e41-a84e-0256209180b7-cert\") pod \"ingress-canary-fnfbq\" (UID: \"d4599713-5ceb-4e41-a84e-0256209180b7\") " pod="openshift-ingress-canary/ingress-canary-fnfbq"
Sep 08 16:13:20.438101 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.438081    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfrnc\" (UniqueName: \"kubernetes.io/projected/ed691713-c786-4c14-b382-8bdedeb4073d-kube-api-access-dfrnc\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.440438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.440418    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvss\" (UniqueName: \"kubernetes.io/projected/d4599713-5ceb-4e41-a84e-0256209180b7-kube-api-access-8zvss\") pod \"ingress-canary-fnfbq\" (UID: \"d4599713-5ceb-4e41-a84e-0256209180b7\") " pod="openshift-ingress-canary/ingress-canary-fnfbq"
Sep 08 16:13:20.443563 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.443190    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-installation-pull-secrets\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.443782 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.443761    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-image-registry-private-configuration\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.445126 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.445105    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-bound-sa-token\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.445556 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.445535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6bxv\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-kube-api-access-z6bxv\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.558053 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.558030    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-fnfbq"
Sep 08 16:13:20.864028 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.863996    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-fnfbq"]
Sep 08 16:13:20.868077 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:13:20.868047    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4599713_5ceb_4e41_a84e_0256209180b7.slice/crio-b00be4ff6cd4e39a661b1ad993afaf90d0f678becba159d59f53929027b48616 WatchSource:0}: Error finding container b00be4ff6cd4e39a661b1ad993afaf90d0f678becba159d59f53929027b48616: Status 404 returned error can't find the container with id b00be4ff6cd4e39a661b1ad993afaf90d0f678becba159d59f53929027b48616
Sep 08 16:13:20.936848 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.936667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:20.936988 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:20.936873    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:20.936988 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.936817    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:13:20.936988 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.936938    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:13:20.937115 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.937003    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:13:21.936984285 +0000 UTC m=+36.023648033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:13:20.937115 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.937044    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:20.937115 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:20.937093    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:13:21.937080168 +0000 UTC m=+36.023743894 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:21.500225 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.500192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:21.500468 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.500203    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:21.500468 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.500311    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:21.503012 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.502987    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 16:13:21.503928 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.503905    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 16:13:21.504054 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.503928    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74mht\""
Sep 08 16:13:21.504054 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.503936    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 16:13:21.504054 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.503990    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 16:13:21.504218 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.504098    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5rq8v\""
Sep 08 16:13:21.698363 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.698329    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fnfbq" event={"ID":"d4599713-5ceb-4e41-a84e-0256209180b7","Type":"ContainerStarted","Data":"b00be4ff6cd4e39a661b1ad993afaf90d0f678becba159d59f53929027b48616"}
Sep 08 16:13:21.945130 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.945094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:21.945130 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:21.945135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:21.945467 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:21.945261    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:21.945467 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:21.945270    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:13:21.945467 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:21.945292    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:13:21.945467 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:21.945314    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:13:23.945296514 +0000 UTC m=+38.031960259 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:21.945467 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:21.945346    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:13:23.945327029 +0000 UTC m=+38.031990764 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:13:23.706889 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:23.706856    2575 generic.go:356] "Generic (PLEG): container finished" podID="0cab0104-2fcc-4735-a30b-b39095652998" containerID="cc1d2410bf3af90e9ab3acf51ce1396df50f279661aa5e392b8452ca5853b8e1" exitCode=0
Sep 08 16:13:23.707502 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:23.706921    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerDied","Data":"cc1d2410bf3af90e9ab3acf51ce1396df50f279661aa5e392b8452ca5853b8e1"}
Sep 08 16:13:23.961851 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:23.961769    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:23.961851 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:23.961814    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:23.962038 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:23.961927    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:13:23.962038 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:23.961949    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:13:23.962038 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:23.961959    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:23.962038 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:23.962012    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:13:27.961996411 +0000 UTC m=+42.048660139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:23.962038 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:23.962026    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:13:27.962020054 +0000 UTC m=+42.048683780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:13:24.713305 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:24.713267    2575 generic.go:356] "Generic (PLEG): container finished" podID="0cab0104-2fcc-4735-a30b-b39095652998" containerID="8fc69e91224b6fbea0331f20674934550f9f8b2358fe75ef58c3d8472c687977" exitCode=0
Sep 08 16:13:24.713846 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:24.713351    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerDied","Data":"8fc69e91224b6fbea0331f20674934550f9f8b2358fe75ef58c3d8472c687977"}
Sep 08 16:13:25.718998 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:25.718966    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lp26m" event={"ID":"0cab0104-2fcc-4735-a30b-b39095652998","Type":"ContainerStarted","Data":"8ba2ab4b2b28c34a12639cb9578b623334f044b4801c38dc2f48849ee710a31a"}
Sep 08 16:13:25.720455 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:25.720434    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-fnfbq" event={"ID":"d4599713-5ceb-4e41-a84e-0256209180b7","Type":"ContainerStarted","Data":"315d8d387fff09e681bbe884a60d110540bc0620df181d8dd2fed04ad5db65d5"}
Sep 08 16:13:25.741884 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:25.741837    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lp26m" podStartSLOduration=8.351117356 podStartE2EDuration="38.741820769s" totalImagesPullingTime="30.390703413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:25.740487204 +0000 UTC m=+39.827150976" watchObservedRunningTime="2026-09-08 16:13:25.741820769 +0000 UTC m=+39.828484517"
Sep 08 16:13:25.758292 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:25.758255    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-fnfbq" podStartSLOduration=1.256090474 podStartE2EDuration="5.758242489s" totalImagesPullingTime="4.502152015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:25.758136845 +0000 UTC m=+39.844800593" watchObservedRunningTime="2026-09-08 16:13:25.758242489 +0000 UTC m=+39.844906237"
Sep 08 16:13:27.992229 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:27.992197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:27.992697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:27.992235    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:27.992697 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:27.992340    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:13:27.992697 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:27.992358    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:13:27.992697 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:27.992378    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:27.992697 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:27.992411    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:13:35.992394865 +0000 UTC m=+50.079058606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:13:27.992697 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:27.992425    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:13:35.992418836 +0000 UTC m=+50.079082562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:36.049679 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:36.049640    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:36.049679 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:36.049677    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:36.050138 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:36.049784    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:36.050138 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:36.049801    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:13:36.050138 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:36.049824    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:13:36.050138 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:36.049848    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:13:52.049831423 +0000 UTC m=+66.136495152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:36.050138 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:36.049891    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:13:52.049873806 +0000 UTC m=+66.136537545 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:13:45.689775 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:45.689747    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bbgqp"
Sep 08 16:13:47.077477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.077444    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"]
Sep 08 16:13:47.079828 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.079812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.083266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.083235    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 08 16:13:47.083266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.083257    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 08 16:13:47.083266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.083238    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 08 16:13:47.083500 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.083238    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 08 16:13:47.089966 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.089943    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"]
Sep 08 16:13:47.129776 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.129751    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-tmp\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.129873 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.129780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv2cr\" (UniqueName: \"kubernetes.io/projected/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-kube-api-access-zv2cr\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.129873 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.129868    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-klusterlet-config\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.230796 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.230752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-klusterlet-config\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.230944 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.230810    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-tmp\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.230944 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.230839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv2cr\" (UniqueName: \"kubernetes.io/projected/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-kube-api-access-zv2cr\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.231207 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.231186    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-tmp\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.233967 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.233944    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-klusterlet-config\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.238177 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.238154    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv2cr\" (UniqueName: \"kubernetes.io/projected/9eb7eb53-77fc-4799-bc28-0f5d15ec2952-kube-api-access-zv2cr\") pod \"klusterlet-addon-workmgr-ffccc8756-sjn97\" (UID: \"9eb7eb53-77fc-4799-bc28-0f5d15ec2952\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.388666 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.388576    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:47.651414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.651329    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"]
Sep 08 16:13:47.654696 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:13:47.654666    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb7eb53_77fc_4799_bc28_0f5d15ec2952.slice/crio-66c4edeae5231d63cbb27dcd53463cbfe9614ee8a767a049b8102d48e3b58b6f WatchSource:0}: Error finding container 66c4edeae5231d63cbb27dcd53463cbfe9614ee8a767a049b8102d48e3b58b6f: Status 404 returned error can't find the container with id 66c4edeae5231d63cbb27dcd53463cbfe9614ee8a767a049b8102d48e3b58b6f
Sep 08 16:13:47.774419 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:47.774384    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" event={"ID":"9eb7eb53-77fc-4799-bc28-0f5d15ec2952","Type":"ContainerStarted","Data":"66c4edeae5231d63cbb27dcd53463cbfe9614ee8a767a049b8102d48e3b58b6f"}
Sep 08 16:13:52.070290 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.070250    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:13:52.070764 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.070296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:13:52.070764 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.070329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:13:52.070764 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.070375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:52.070764 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:52.070425    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:13:52.070764 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:52.070434    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:13:52.070764 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:52.070459    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:13:52.070764 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:52.070494    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:14:24.070472157 +0000 UTC m=+98.157135888 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:13:52.070764 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:52.070523    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:14:24.070504723 +0000 UTC m=+98.157168468 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:13:52.073015 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.072995    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 08 16:13:52.073138 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.073064    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 08 16:13:52.081323 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:52.081305    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 16:13:52.081422 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:13:52.081409    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:14:56.08139153 +0000 UTC m=+130.168055263 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : secret "metrics-daemon-secret" not found
Sep 08 16:13:52.083647 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.083628    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4133bb28-b1e2-4b78-9dc3-40043ee78b17-original-pull-secret\") pod \"global-pull-secret-syncer-fdnml\" (UID: \"4133bb28-b1e2-4b78-9dc3-40043ee78b17\") " pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:52.118234 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.118211    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fdnml"
Sep 08 16:13:52.170923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.170898    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:52.173722 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.173703    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 08 16:13:52.183649 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.183628    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 08 16:13:52.195405 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.195373    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hv77\" (UniqueName: \"kubernetes.io/projected/77c33fb0-b890-4124-a87d-2acf5ef7ce83-kube-api-access-4hv77\") pod \"network-check-target-vl6bn\" (UID: \"77c33fb0-b890-4124-a87d-2acf5ef7ce83\") " pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:52.414423 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.414397    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-74mht\""
Sep 08 16:13:52.423163 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.423137    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:52.657783 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.657710    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fdnml"]
Sep 08 16:13:52.660314 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:13:52.660289    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4133bb28_b1e2_4b78_9dc3_40043ee78b17.slice/crio-758cc7d37ae1773206690179e69a9eb4e27473d0237e3b72735843100dbaafb7 WatchSource:0}: Error finding container 758cc7d37ae1773206690179e69a9eb4e27473d0237e3b72735843100dbaafb7: Status 404 returned error can't find the container with id 758cc7d37ae1773206690179e69a9eb4e27473d0237e3b72735843100dbaafb7
Sep 08 16:13:52.694339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.694318    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vl6bn"]
Sep 08 16:13:52.696810 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:13:52.696784    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c33fb0_b890_4124_a87d_2acf5ef7ce83.slice/crio-0c239a476da737564b5d07124f4d9cfbe257503ae863bbe228de3f29dde1e4f6 WatchSource:0}: Error finding container 0c239a476da737564b5d07124f4d9cfbe257503ae863bbe228de3f29dde1e4f6: Status 404 returned error can't find the container with id 0c239a476da737564b5d07124f4d9cfbe257503ae863bbe228de3f29dde1e4f6
Sep 08 16:13:52.789249 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.789224    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" event={"ID":"9eb7eb53-77fc-4799-bc28-0f5d15ec2952","Type":"ContainerStarted","Data":"37db98e81c075388eae11c65b6c293ab03c854494c2d7bee6d2dfea9c3ee34e2"}
Sep 08 16:13:52.789458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.789440    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:52.790626 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.790604    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vl6bn" event={"ID":"77c33fb0-b890-4124-a87d-2acf5ef7ce83","Type":"ContainerStarted","Data":"0c239a476da737564b5d07124f4d9cfbe257503ae863bbe228de3f29dde1e4f6"}
Sep 08 16:13:52.790931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.790910    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:13:52.791743 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.791724    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fdnml" event={"ID":"4133bb28-b1e2-4b78-9dc3-40043ee78b17","Type":"ContainerStarted","Data":"758cc7d37ae1773206690179e69a9eb4e27473d0237e3b72735843100dbaafb7"}
Sep 08 16:13:52.803324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:52.803287    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" podStartSLOduration=1.011827058 podStartE2EDuration="5.803275691s" totalImagesPullingTime="4.791448633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:52.802949404 +0000 UTC m=+66.889613166" watchObservedRunningTime="2026-09-08 16:13:52.803275691 +0000 UTC m=+66.889939438"
Sep 08 16:13:57.807358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:57.807324    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vl6bn" event={"ID":"77c33fb0-b890-4124-a87d-2acf5ef7ce83","Type":"ContainerStarted","Data":"3932f757423701fece759af423bde9ec8490314432463df7652a33632d643def"}
Sep 08 16:13:57.807724 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:57.807387    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:13:57.808854 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:57.808831    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fdnml" event={"ID":"4133bb28-b1e2-4b78-9dc3-40043ee78b17","Type":"ContainerStarted","Data":"938eb2e0e1f6c0349e0c7342369fd561d995864435df5b8ef735edc156388103"}
Sep 08 16:13:57.823163 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:57.823123    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vl6bn" podStartSLOduration=65.910388985 podStartE2EDuration="1m10.823109613s" totalImagesPullingTime="4.912720628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:57.822461338 +0000 UTC m=+71.909125087" watchObservedRunningTime="2026-09-08 16:13:57.823109613 +0000 UTC m=+71.909773358"
Sep 08 16:13:57.837715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:13:57.837662    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-fdnml" podStartSLOduration=65.877750556 podStartE2EDuration="1m10.837649432s" totalImagesPullingTime="4.959898876s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:13:57.837645901 +0000 UTC m=+71.924309650" watchObservedRunningTime="2026-09-08 16:13:57.837649432 +0000 UTC m=+71.924313222"
Sep 08 16:14:24.083298 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:14:24.083263    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:14:24.083298 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:14:24.083300    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:14:24.083778 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:14:24.083415    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:14:24.083778 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:14:24.083465    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:15:28.083451438 +0000 UTC m=+162.170115177 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:14:24.083778 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:14:24.083419    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:14:24.083778 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:14:24.083488    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:14:24.083778 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:14:24.083522    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:15:28.083512605 +0000 UTC m=+162.170176347 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:14:28.813041 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:14:28.813006    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vl6bn"
Sep 08 16:14:56.098660 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:14:56.098610    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:14:56.099177 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:14:56.098750    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 08 16:14:56.099177 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:14:56.098836    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs podName:0d770632-49ab-40c5-87f5-4420e74a1a44 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:58.098817975 +0000 UTC m=+252.185481706 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs") pod "network-metrics-daemon-d7gtt" (UID: "0d770632-49ab-40c5-87f5-4420e74a1a44") : secret "metrics-daemon-secret" not found
Sep 08 16:15:23.214517 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:15:23.214481    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" podUID="62641f8d-0486-444e-b887-a345668b437f"
Sep 08 16:15:23.231989 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:15:23.231967    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-sl68l" podUID="ed691713-c786-4c14-b382-8bdedeb4073d"
Sep 08 16:15:24.019628 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:24.019601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sl68l"
Sep 08 16:15:24.019790 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:24.019601    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:15:24.524432 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:15:24.524390    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-d7gtt" podUID="0d770632-49ab-40c5-87f5-4420e74a1a44"
Sep 08 16:15:28.115666 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:28.115624    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:15:28.115666 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:28.115667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:15:28.116073 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:15:28.115759    2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 08 16:15:28.116073 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:15:28.115765    2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 08 16:15:28.116073 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:15:28.115780    2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c595d89b8-mxfw5: secret "image-registry-tls" not found
Sep 08 16:15:28.116073 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:15:28.115816    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls podName:ed691713-c786-4c14-b382-8bdedeb4073d nodeName:}" failed. No retries permitted until 2026-09-08 16:17:30.115802852 +0000 UTC m=+284.202466582 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls") pod "dns-default-sl68l" (UID: "ed691713-c786-4c14-b382-8bdedeb4073d") : secret "dns-default-metrics-tls" not found
Sep 08 16:15:28.116073 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:15:28.115835    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls podName:62641f8d-0486-444e-b887-a345668b437f nodeName:}" failed. No retries permitted until 2026-09-08 16:17:30.115821937 +0000 UTC m=+284.202485675 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls") pod "image-registry-6c595d89b8-mxfw5" (UID: "62641f8d-0486-444e-b887-a345668b437f") : secret "image-registry-tls" not found
Sep 08 16:15:36.501481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:36.501448    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:15:52.789953 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:52.789861    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" podUID="9eb7eb53-77fc-4799-bc28-0f5d15ec2952" containerName="acm-agent" probeResult="failure" output="Get \"http://10.133.0.9:8000/readyz\": dial tcp 10.133.0.9:8000: connect: connection refused"
Sep 08 16:15:53.090047 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:53.090016    2575 generic.go:356] "Generic (PLEG): container finished" podID="9eb7eb53-77fc-4799-bc28-0f5d15ec2952" containerID="37db98e81c075388eae11c65b6c293ab03c854494c2d7bee6d2dfea9c3ee34e2" exitCode=1
Sep 08 16:15:53.090195 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:53.090095    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" event={"ID":"9eb7eb53-77fc-4799-bc28-0f5d15ec2952","Type":"ContainerDied","Data":"37db98e81c075388eae11c65b6c293ab03c854494c2d7bee6d2dfea9c3ee34e2"}
Sep 08 16:15:53.090434 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:53.090417    2575 scope.go:117] "RemoveContainer" containerID="37db98e81c075388eae11c65b6c293ab03c854494c2d7bee6d2dfea9c3ee34e2"
Sep 08 16:15:54.094527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:54.094494    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" event={"ID":"9eb7eb53-77fc-4799-bc28-0f5d15ec2952","Type":"ContainerStarted","Data":"2f697af2b51cbcb5fa0ed1c84d809c95d72985ed5fb8e4c84de3a55bc13f0a8d"}
Sep 08 16:15:54.094900 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:54.094753    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:15:54.098669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:15:54.098649    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:16:34.714315 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.714277    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-72szv"]
Sep 08 16:16:34.716427 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.716405    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.722773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.722415    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 08 16:16:34.723445 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.723420    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 08 16:16:34.723710 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.723682    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 08 16:16:34.724003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.723963    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 08 16:16:34.724003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.723980    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-h6vvs\""
Sep 08 16:16:34.724155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.724015    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gmmnn"]
Sep 08 16:16:34.725866 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.725849    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-gmmnn"
Sep 08 16:16:34.728941 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.728921    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-cb9pl\""
Sep 08 16:16:34.734447 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.734427    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 08 16:16:34.738230 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.738205    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-72szv"]
Sep 08 16:16:34.749699 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.749670    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gmmnn"]
Sep 08 16:16:34.817768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.817749    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-849b99ddd-v5nhz"]
Sep 08 16:16:34.819396 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.819382    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:34.830933 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.830916    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Sep 08 16:16:34.831715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.831699    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 08 16:16:34.831799 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.831740    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 08 16:16:34.832176 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.832160    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Sep 08 16:16:34.832252 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.832194    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Sep 08 16:16:34.832410 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.832387    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Sep 08 16:16:34.832482 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.832390    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wjl25\""
Sep 08 16:16:34.850041 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.850022    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rbwq\" (UniqueName: \"kubernetes.io/projected/48e2b9dd-5f3e-44a5-86b3-7977c68faf69-kube-api-access-5rbwq\") pod \"network-check-source-685456655f-gmmnn\" (UID: \"48e2b9dd-5f3e-44a5-86b3-7977c68faf69\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gmmnn"
Sep 08 16:16:34.850101 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.850051    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4fce580-bb38-47e7-880f-90cafa36b3ec-trusted-ca\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.850101 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.850087    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4fce580-bb38-47e7-880f-90cafa36b3ec-config\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.850178 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.850123    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gdjb\" (UniqueName: \"kubernetes.io/projected/a4fce580-bb38-47e7-880f-90cafa36b3ec-kube-api-access-5gdjb\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.850178 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.850148    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4fce580-bb38-47e7-880f-90cafa36b3ec-serving-cert\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.865052 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.865032    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-849b99ddd-v5nhz"]
Sep 08 16:16:34.951277 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951253    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhtwf\" (UniqueName: \"kubernetes.io/projected/28efe48a-f884-4d00-a8dd-f33b1a14bae4-kube-api-access-hhtwf\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:34.951361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951286    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4fce580-bb38-47e7-880f-90cafa36b3ec-config\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.951361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gdjb\" (UniqueName: \"kubernetes.io/projected/a4fce580-bb38-47e7-880f-90cafa36b3ec-kube-api-access-5gdjb\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.951361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951324    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:34.951361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951346    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4fce580-bb38-47e7-880f-90cafa36b3ec-serving-cert\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.951487 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951378    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:34.951554 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-stats-auth\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:34.951627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951608    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rbwq\" (UniqueName: \"kubernetes.io/projected/48e2b9dd-5f3e-44a5-86b3-7977c68faf69-kube-api-access-5rbwq\") pod \"network-check-source-685456655f-gmmnn\" (UID: \"48e2b9dd-5f3e-44a5-86b3-7977c68faf69\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gmmnn"
Sep 08 16:16:34.951672 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951639    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4fce580-bb38-47e7-880f-90cafa36b3ec-trusted-ca\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.951672 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.951655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-default-certificate\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:34.952036 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.952020    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4fce580-bb38-47e7-880f-90cafa36b3ec-config\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.952319 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.952302    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4fce580-bb38-47e7-880f-90cafa36b3ec-trusted-ca\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.953524 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.953507    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4fce580-bb38-47e7-880f-90cafa36b3ec-serving-cert\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:34.960277 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.960251    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rbwq\" (UniqueName: \"kubernetes.io/projected/48e2b9dd-5f3e-44a5-86b3-7977c68faf69-kube-api-access-5rbwq\") pod \"network-check-source-685456655f-gmmnn\" (UID: \"48e2b9dd-5f3e-44a5-86b3-7977c68faf69\") " pod="openshift-network-diagnostics/network-check-source-685456655f-gmmnn"
Sep 08 16:16:34.960567 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:34.960549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gdjb\" (UniqueName: \"kubernetes.io/projected/a4fce580-bb38-47e7-880f-90cafa36b3ec-kube-api-access-5gdjb\") pod \"console-operator-85b454874b-72szv\" (UID: \"a4fce580-bb38-47e7-880f-90cafa36b3ec\") " pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:35.026164 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.026116    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:35.033891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.033868    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-gmmnn"
Sep 08 16:16:35.052832 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.052805    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-default-certificate\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.052917 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.052856    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhtwf\" (UniqueName: \"kubernetes.io/projected/28efe48a-f884-4d00-a8dd-f33b1a14bae4-kube-api-access-hhtwf\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.052917 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.052897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.052993 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.052922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.053036 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:35.053011    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 16:16:35.053083 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:35.053067    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:35.553047787 +0000 UTC m=+229.639711515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : secret "router-metrics-certs-default" not found
Sep 08 16:16:35.053127 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:35.053085    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:35.553075352 +0000 UTC m=+229.639739077 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : configmap references non-existent config key: service-ca.crt
Sep 08 16:16:35.053127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.053103    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-stats-auth\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.055285 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.055265    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-stats-auth\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.055724 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.055705    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-default-certificate\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.063668 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.063643    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhtwf\" (UniqueName: \"kubernetes.io/projected/28efe48a-f884-4d00-a8dd-f33b1a14bae4-kube-api-access-hhtwf\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.300118 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.300045    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-72szv"]
Sep 08 16:16:35.302986 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:16:35.302961    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fce580_bb38_47e7_880f_90cafa36b3ec.slice/crio-171d2d24e5b4cc64154823bac45bf0ea2630843414d156bac0ae5a0a69365492 WatchSource:0}: Error finding container 171d2d24e5b4cc64154823bac45bf0ea2630843414d156bac0ae5a0a69365492: Status 404 returned error can't find the container with id 171d2d24e5b4cc64154823bac45bf0ea2630843414d156bac0ae5a0a69365492
Sep 08 16:16:35.306124 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.306104    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-gmmnn"]
Sep 08 16:16:35.308913 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:16:35.308892    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e2b9dd_5f3e_44a5_86b3_7977c68faf69.slice/crio-22c6f8eb83a9480b687244b84d0ae349fba8620206793f87b864c66ceb953c2b WatchSource:0}: Error finding container 22c6f8eb83a9480b687244b84d0ae349fba8620206793f87b864c66ceb953c2b: Status 404 returned error can't find the container with id 22c6f8eb83a9480b687244b84d0ae349fba8620206793f87b864c66ceb953c2b
Sep 08 16:16:35.557574 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.557513    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.557574 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:35.557549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:35.557736 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:35.557706    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:36.55768784 +0000 UTC m=+230.644351587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : configmap references non-existent config key: service-ca.crt
Sep 08 16:16:35.557776 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:35.557733    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 16:16:35.557776 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:35.557771    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:36.557757866 +0000 UTC m=+230.644421606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : secret "router-metrics-certs-default" not found
Sep 08 16:16:36.220032 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:36.219999    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-gmmnn" event={"ID":"48e2b9dd-5f3e-44a5-86b3-7977c68faf69","Type":"ContainerStarted","Data":"6cfe6c195a8fb896d8b04696a9b47bf40046796dc4dcf6a59f0bc64d35a95a1f"}
Sep 08 16:16:36.220448 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:36.220048    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-gmmnn" event={"ID":"48e2b9dd-5f3e-44a5-86b3-7977c68faf69","Type":"ContainerStarted","Data":"22c6f8eb83a9480b687244b84d0ae349fba8620206793f87b864c66ceb953c2b"}
Sep 08 16:16:36.221169 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:36.221147    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-72szv" event={"ID":"a4fce580-bb38-47e7-880f-90cafa36b3ec","Type":"ContainerStarted","Data":"171d2d24e5b4cc64154823bac45bf0ea2630843414d156bac0ae5a0a69365492"}
Sep 08 16:16:36.234396 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:36.234319    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-gmmnn" podStartSLOduration=2.234304894 podStartE2EDuration="2.234304894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:36.234211478 +0000 UTC m=+230.320875227" watchObservedRunningTime="2026-09-08 16:16:36.234304894 +0000 UTC m=+230.320968642"
Sep 08 16:16:36.565065 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:36.564982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:36.565065 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:36.565030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:36.565267 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:36.565137    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 16:16:36.565267 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:36.565192    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:38.565175022 +0000 UTC m=+232.651838767 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : secret "router-metrics-certs-default" not found
Sep 08 16:16:36.565267 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:36.565235    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:38.565222978 +0000 UTC m=+232.651886710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : configmap references non-existent config key: service-ca.crt
Sep 08 16:16:38.228564 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:38.228538    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/0.log"
Sep 08 16:16:38.228919 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:38.228606    2575 generic.go:356] "Generic (PLEG): container finished" podID="a4fce580-bb38-47e7-880f-90cafa36b3ec" containerID="448a589854a786c0b6cdac948996e4e7778990f4af81528eb7e3a9508e2e0120" exitCode=255
Sep 08 16:16:38.228919 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:38.228675    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-72szv" event={"ID":"a4fce580-bb38-47e7-880f-90cafa36b3ec","Type":"ContainerDied","Data":"448a589854a786c0b6cdac948996e4e7778990f4af81528eb7e3a9508e2e0120"}
Sep 08 16:16:38.228919 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:38.228856    2575 scope.go:117] "RemoveContainer" containerID="448a589854a786c0b6cdac948996e4e7778990f4af81528eb7e3a9508e2e0120"
Sep 08 16:16:38.581326 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:38.581296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:38.581450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:38.581333    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:38.581525 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:38.581452    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 16:16:38.581525 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:38.581502    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:42.581474893 +0000 UTC m=+236.668138651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : configmap references non-existent config key: service-ca.crt
Sep 08 16:16:38.581646 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:38.581536    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:42.581523189 +0000 UTC m=+236.668186915 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : secret "router-metrics-certs-default" not found
Sep 08 16:16:39.233598 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.233552    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/1.log"
Sep 08 16:16:39.234020 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.234001    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/0.log"
Sep 08 16:16:39.234070 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.234040    2575 generic.go:356] "Generic (PLEG): container finished" podID="a4fce580-bb38-47e7-880f-90cafa36b3ec" containerID="24df90a096cd98d059ea5d6ba93ec00928c2a1e40b6ac24fe4120cc753ddcb81" exitCode=255
Sep 08 16:16:39.234135 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.234073    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-72szv" event={"ID":"a4fce580-bb38-47e7-880f-90cafa36b3ec","Type":"ContainerDied","Data":"24df90a096cd98d059ea5d6ba93ec00928c2a1e40b6ac24fe4120cc753ddcb81"}
Sep 08 16:16:39.234187 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.234135    2575 scope.go:117] "RemoveContainer" containerID="448a589854a786c0b6cdac948996e4e7778990f4af81528eb7e3a9508e2e0120"
Sep 08 16:16:39.234376 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.234358    2575 scope.go:117] "RemoveContainer" containerID="24df90a096cd98d059ea5d6ba93ec00928c2a1e40b6ac24fe4120cc753ddcb81"
Sep 08 16:16:39.234565 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:39.234543    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-72szv_openshift-console-operator(a4fce580-bb38-47e7-880f-90cafa36b3ec)\"" pod="openshift-console-operator/console-operator-85b454874b-72szv" podUID="a4fce580-bb38-47e7-880f-90cafa36b3ec"
Sep 08 16:16:39.711293 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.711261    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rjvfx"]
Sep 08 16:16:39.713913 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.713896    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.717256 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.717231    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 08 16:16:39.717505 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.717480    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mlgpx\""
Sep 08 16:16:39.718252 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.718235    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 08 16:16:39.718339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.718274    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 08 16:16:39.718918 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.718893    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 08 16:16:39.726830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.726811    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rjvfx"]
Sep 08 16:16:39.789824 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.789801    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ede55c97-e9a4-4599-95c5-9b35ca349745-data-volume\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.789921 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.789853    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.789986 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.789949    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ede55c97-e9a4-4599-95c5-9b35ca349745-crio-socket\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.790041 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.790001    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xx2l\" (UniqueName: \"kubernetes.io/projected/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-api-access-8xx2l\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.790094 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.790054    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.891266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.891243    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.891450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.891279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ede55c97-e9a4-4599-95c5-9b35ca349745-crio-socket\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.891450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.891305    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xx2l\" (UniqueName: \"kubernetes.io/projected/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-api-access-8xx2l\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.891450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.891346    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.891450 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:39.891372    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:39.891450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.891382    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ede55c97-e9a4-4599-95c5-9b35ca349745-data-volume\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.891450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.891390    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ede55c97-e9a4-4599-95c5-9b35ca349745-crio-socket\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.891450 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:39.891440    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:40.391419756 +0000 UTC m=+234.478083497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:39.891825 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:39.891498    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:39.891825 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:39.891554    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:40.391538843 +0000 UTC m=+234.478202585 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:39.891825 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.891652    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ede55c97-e9a4-4599-95c5-9b35ca349745-data-volume\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:39.900438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:39.900418    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xx2l\" (UniqueName: \"kubernetes.io/projected/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-api-access-8xx2l\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:40.238459 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:40.238437    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/1.log"
Sep 08 16:16:40.238777 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:40.238741    2575 scope.go:117] "RemoveContainer" containerID="24df90a096cd98d059ea5d6ba93ec00928c2a1e40b6ac24fe4120cc753ddcb81"
Sep 08 16:16:40.238900 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:40.238880    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-72szv_openshift-console-operator(a4fce580-bb38-47e7-880f-90cafa36b3ec)\"" pod="openshift-console-operator/console-operator-85b454874b-72szv" podUID="a4fce580-bb38-47e7-880f-90cafa36b3ec"
Sep 08 16:16:40.394632 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:40.394606    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:40.394746 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:40.394664    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:40.394746 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:40.394720    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:40.394746 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:40.394729    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:40.394858 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:40.394759    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:41.394745899 +0000 UTC m=+235.481409625 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:40.394858 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:40.394778    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:41.394764305 +0000 UTC m=+235.481428033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:41.402910 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:41.402877    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:41.403294 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:41.402943    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:41.403294 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:41.403009    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:41.403294 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:41.403010    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:41.403294 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:41.403062    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:43.403048478 +0000 UTC m=+237.489712204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:41.403294 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:41.403076    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:43.403069492 +0000 UTC m=+237.489733218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:42.364294 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.364267    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9l8gd_ef6c43f1-b4e3-464f-a4ab-57c86138262b/dns-node-resolver/0.log"
Sep 08 16:16:42.397941 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.397914    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-csqx9"]
Sep 08 16:16:42.399755 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.399738    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.402169 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.402143    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 08 16:16:42.402274 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.402220    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 08 16:16:42.402274 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.402235    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-9skhc\""
Sep 08 16:16:42.403271 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.403248    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 08 16:16:42.403271 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.403258    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 08 16:16:42.409678 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.409656    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-csqx9"]
Sep 08 16:16:42.509288 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.509259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27zzm\" (UniqueName: \"kubernetes.io/projected/53331f0f-f7e6-4c32-a8d8-e4209070414f-kube-api-access-27zzm\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.509394 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.509297    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/53331f0f-f7e6-4c32-a8d8-e4209070414f-signing-key\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.509394 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.509365    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/53331f0f-f7e6-4c32-a8d8-e4209070414f-signing-cabundle\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.610222 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.610197    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:42.610320 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.610247    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/53331f0f-f7e6-4c32-a8d8-e4209070414f-signing-cabundle\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.610320 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.610268    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:42.610447 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:42.610349    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 16:16:42.610447 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:42.610391    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:50.610367643 +0000 UTC m=+244.697031382 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : configmap references non-existent config key: service-ca.crt
Sep 08 16:16:42.610447 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:42.610420    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:50.610407418 +0000 UTC m=+244.697071146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : secret "router-metrics-certs-default" not found
Sep 08 16:16:42.610635 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.610530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27zzm\" (UniqueName: \"kubernetes.io/projected/53331f0f-f7e6-4c32-a8d8-e4209070414f-kube-api-access-27zzm\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.610635 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.610572    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/53331f0f-f7e6-4c32-a8d8-e4209070414f-signing-key\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.611078 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.611061    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/53331f0f-f7e6-4c32-a8d8-e4209070414f-signing-cabundle\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.612940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.612924    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/53331f0f-f7e6-4c32-a8d8-e4209070414f-signing-key\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.618262 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.618210    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27zzm\" (UniqueName: \"kubernetes.io/projected/53331f0f-f7e6-4c32-a8d8-e4209070414f-kube-api-access-27zzm\") pod \"service-ca-75d99c484-csqx9\" (UID: \"53331f0f-f7e6-4c32-a8d8-e4209070414f\") " pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.709429 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.709403    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-csqx9"
Sep 08 16:16:42.973496 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:42.973432    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-csqx9"]
Sep 08 16:16:42.976289 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:16:42.976264    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53331f0f_f7e6_4c32_a8d8_e4209070414f.slice/crio-f001fce26416089a3a00c0fc3811131e38b6dddd72b6a171e07fba25edced5a8 WatchSource:0}: Error finding container f001fce26416089a3a00c0fc3811131e38b6dddd72b6a171e07fba25edced5a8: Status 404 returned error can't find the container with id f001fce26416089a3a00c0fc3811131e38b6dddd72b6a171e07fba25edced5a8
Sep 08 16:16:43.248632 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:43.248547    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-csqx9" event={"ID":"53331f0f-f7e6-4c32-a8d8-e4209070414f","Type":"ContainerStarted","Data":"f001fce26416089a3a00c0fc3811131e38b6dddd72b6a171e07fba25edced5a8"}
Sep 08 16:16:43.365271 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:43.365249    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxwz6_6f8613b9-7b07-468f-a48c-ebb3d4d519e4/node-ca/0.log"
Sep 08 16:16:43.417786 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:43.417756    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:43.418076 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:43.417828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:43.418076 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:43.417887    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:43.418076 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:43.417955    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:47.41793591 +0000 UTC m=+241.504599643 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:43.418076 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:43.417959    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:43.418076 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:43.417994    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:47.417983838 +0000 UTC m=+241.504647578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:44.365104 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:44.365076    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fnfbq_d4599713-5ceb-4e41-a84e-0256209180b7/serve-healthcheck-canary/0.log"
Sep 08 16:16:45.027014 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:45.026941    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:45.027014 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:45.027007    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:16:45.027487 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:45.027338    2575 scope.go:117] "RemoveContainer" containerID="24df90a096cd98d059ea5d6ba93ec00928c2a1e40b6ac24fe4120cc753ddcb81"
Sep 08 16:16:45.027547 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:45.027502    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-72szv_openshift-console-operator(a4fce580-bb38-47e7-880f-90cafa36b3ec)\"" pod="openshift-console-operator/console-operator-85b454874b-72szv" podUID="a4fce580-bb38-47e7-880f-90cafa36b3ec"
Sep 08 16:16:46.259772 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:46.259690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-csqx9" event={"ID":"53331f0f-f7e6-4c32-a8d8-e4209070414f","Type":"ContainerStarted","Data":"e057e154bfc3c7cc3b22075a61c8dfb6347416db835e35c5fb67c099f8a68e28"}
Sep 08 16:16:46.279956 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:46.279903    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-csqx9" podStartSLOduration=1.7840002190000002 podStartE2EDuration="4.2798835s" totalImagesPullingTime="2.495883281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:16:46.279464367 +0000 UTC m=+240.366128115" watchObservedRunningTime="2026-09-08 16:16:46.2798835 +0000 UTC m=+240.366547250"
Sep 08 16:16:47.445324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:47.445289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:47.445716 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:47.445410    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:47.445716 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:47.445453    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:47.445716 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:47.445464    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:55.445450435 +0000 UTC m=+249.532114187 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:47.445716 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:47.445555    2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 08 16:16:47.445716 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:47.445610    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:16:55.445599196 +0000 UTC m=+249.532262921 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : configmap "kube-rbac-proxy" not found
Sep 08 16:16:50.667414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:50.667384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:50.667793 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:50.667464    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:16:50.667793 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:50.667541    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:17:06.667523562 +0000 UTC m=+260.754187288 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : configmap references non-existent config key: service-ca.crt
Sep 08 16:16:50.667793 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:50.667600    2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Sep 08 16:16:50.667793 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:50.667654    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs podName:28efe48a-f884-4d00-a8dd-f33b1a14bae4 nodeName:}" failed. No retries permitted until 2026-09-08 16:17:06.667638583 +0000 UTC m=+260.754302316 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs") pod "router-default-849b99ddd-v5nhz" (UID: "28efe48a-f884-4d00-a8dd-f33b1a14bae4") : secret "router-metrics-certs-default" not found
Sep 08 16:16:55.503455 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:55.503417    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:55.504109 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:55.503488    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:55.504109 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:55.503565    2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:55.504109 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:55.503661    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls podName:ede55c97-e9a4-4599-95c5-9b35ca349745 nodeName:}" failed. No retries permitted until 2026-09-08 16:17:11.503644404 +0000 UTC m=+265.590308133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rjvfx" (UID: "ede55c97-e9a4-4599-95c5-9b35ca349745") : secret "insights-runtime-extractor-tls" not found
Sep 08 16:16:55.504290 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:55.504183    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ede55c97-e9a4-4599-95c5-9b35ca349745-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:16:58.122694 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:58.122657    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:16:58.124949 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:58.124926    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d770632-49ab-40c5-87f5-4420e74a1a44-metrics-certs\") pod \"network-metrics-daemon-d7gtt\" (UID: \"0d770632-49ab-40c5-87f5-4420e74a1a44\") " pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:16:58.404522 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:58.404458    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5rq8v\""
Sep 08 16:16:58.412555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:58.412534    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d7gtt"
Sep 08 16:16:58.501267 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:58.501239    2575 scope.go:117] "RemoveContainer" containerID="24df90a096cd98d059ea5d6ba93ec00928c2a1e40b6ac24fe4120cc753ddcb81"
Sep 08 16:16:58.678272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:58.678221    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d7gtt"]
Sep 08 16:16:58.680711 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:16:58.680690    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d770632_49ab_40c5_87f5_4420e74a1a44.slice/crio-e77811bc78dbf78c3f93a635b8d08fb173e1d97cbbdf5c553fb20c2c27d5d094 WatchSource:0}: Error finding container e77811bc78dbf78c3f93a635b8d08fb173e1d97cbbdf5c553fb20c2c27d5d094: Status 404 returned error can't find the container with id e77811bc78dbf78c3f93a635b8d08fb173e1d97cbbdf5c553fb20c2c27d5d094
Sep 08 16:16:59.300052 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:59.300015    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d7gtt" event={"ID":"0d770632-49ab-40c5-87f5-4420e74a1a44","Type":"ContainerStarted","Data":"e77811bc78dbf78c3f93a635b8d08fb173e1d97cbbdf5c553fb20c2c27d5d094"}
Sep 08 16:16:59.301858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:59.301836    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:16:59.302312 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:59.302294    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/1.log"
Sep 08 16:16:59.302387 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:59.302329    2575 generic.go:356] "Generic (PLEG): container finished" podID="a4fce580-bb38-47e7-880f-90cafa36b3ec" containerID="1d43a1ee9dbaf51fbd1fa48b4cc6cfaa9c3965780d2488a12772bc1d591859a5" exitCode=255
Sep 08 16:16:59.302387 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:59.302372    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-72szv" event={"ID":"a4fce580-bb38-47e7-880f-90cafa36b3ec","Type":"ContainerDied","Data":"1d43a1ee9dbaf51fbd1fa48b4cc6cfaa9c3965780d2488a12772bc1d591859a5"}
Sep 08 16:16:59.302458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:59.302418    2575 scope.go:117] "RemoveContainer" containerID="24df90a096cd98d059ea5d6ba93ec00928c2a1e40b6ac24fe4120cc753ddcb81"
Sep 08 16:16:59.302783 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:16:59.302754    2575 scope.go:117] "RemoveContainer" containerID="1d43a1ee9dbaf51fbd1fa48b4cc6cfaa9c3965780d2488a12772bc1d591859a5"
Sep 08 16:16:59.302956 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:16:59.302935    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-72szv_openshift-console-operator(a4fce580-bb38-47e7-880f-90cafa36b3ec)\"" pod="openshift-console-operator/console-operator-85b454874b-72szv" podUID="a4fce580-bb38-47e7-880f-90cafa36b3ec"
Sep 08 16:17:00.306761 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:00.306739    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:17:00.308537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:00.308510    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d7gtt" event={"ID":"0d770632-49ab-40c5-87f5-4420e74a1a44","Type":"ContainerStarted","Data":"15940dae8a5e268751e975859f5fc8b74bf73acaebc9d323ea2eb4ddd275436f"}
Sep 08 16:17:00.308622 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:00.308604    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d7gtt" event={"ID":"0d770632-49ab-40c5-87f5-4420e74a1a44","Type":"ContainerStarted","Data":"d01df8f409dedb6cc3579400586124cb76cbce67100289e70c3f59bb5efa6f28"}
Sep 08 16:17:00.326483 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:00.326435    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d7gtt" podStartSLOduration=251.916102745 podStartE2EDuration="4m13.326422551s" totalImagesPullingTime="1.410319806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:00.325326958 +0000 UTC m=+254.411990706" watchObservedRunningTime="2026-09-08 16:17:00.326422551 +0000 UTC m=+254.413086299"
Sep 08 16:17:05.027001 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:05.026970    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:17:05.027370 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:05.027025    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:17:05.027370 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:05.027273    2575 scope.go:117] "RemoveContainer" containerID="1d43a1ee9dbaf51fbd1fa48b4cc6cfaa9c3965780d2488a12772bc1d591859a5"
Sep 08 16:17:05.027439 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:17:05.027415    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-72szv_openshift-console-operator(a4fce580-bb38-47e7-880f-90cafa36b3ec)\"" pod="openshift-console-operator/console-operator-85b454874b-72szv" podUID="a4fce580-bb38-47e7-880f-90cafa36b3ec"
Sep 08 16:17:06.684658 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:06.684626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:06.685193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:06.684690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:06.685193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:06.685184    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/28efe48a-f884-4d00-a8dd-f33b1a14bae4-service-ca-bundle\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:06.686974 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:06.686949    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28efe48a-f884-4d00-a8dd-f33b1a14bae4-metrics-certs\") pod \"router-default-849b99ddd-v5nhz\" (UID: \"28efe48a-f884-4d00-a8dd-f33b1a14bae4\") " pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:06.929297 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:06.929269    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-wjl25\""
Sep 08 16:17:06.937685 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:06.937628    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:07.205537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:07.205456    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-849b99ddd-v5nhz"]
Sep 08 16:17:07.208543 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:07.208514    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28efe48a_f884_4d00_a8dd_f33b1a14bae4.slice/crio-f38e23766cabd8d06a378ae7262af392806cebcab3b75b20175abf31816af783 WatchSource:0}: Error finding container f38e23766cabd8d06a378ae7262af392806cebcab3b75b20175abf31816af783: Status 404 returned error can't find the container with id f38e23766cabd8d06a378ae7262af392806cebcab3b75b20175abf31816af783
Sep 08 16:17:07.330481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:07.330453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-849b99ddd-v5nhz" event={"ID":"28efe48a-f884-4d00-a8dd-f33b1a14bae4","Type":"ContainerStarted","Data":"7a53713314a25dcd3ed7254a91dfad187d394b9ec54c9488e1234e39c43041b4"}
Sep 08 16:17:07.330629 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:07.330496    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-849b99ddd-v5nhz" event={"ID":"28efe48a-f884-4d00-a8dd-f33b1a14bae4","Type":"ContainerStarted","Data":"f38e23766cabd8d06a378ae7262af392806cebcab3b75b20175abf31816af783"}
Sep 08 16:17:07.348020 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:07.347976    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-849b99ddd-v5nhz" podStartSLOduration=33.347961811 podStartE2EDuration="33.347961811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:07.347545668 +0000 UTC m=+261.434209429" watchObservedRunningTime="2026-09-08 16:17:07.347961811 +0000 UTC m=+261.434625558"
Sep 08 16:17:07.938627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:07.938594    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:07.940556 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:07.940530    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:08.333737 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:08.333712    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:08.334957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:08.334933    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-849b99ddd-v5nhz"
Sep 08 16:17:09.387879 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.387846    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"]
Sep 08 16:17:09.390877 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.390859    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"
Sep 08 16:17:09.393232 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.393210    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 08 16:17:09.393340 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.393238    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dsjhg\""
Sep 08 16:17:09.399987 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.399964    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"]
Sep 08 16:17:09.504957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.504915    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/707a51d2-c191-4bf0-8444-baa02cc338e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-mmjbq\" (UID: \"707a51d2-c191-4bf0-8444-baa02cc338e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"
Sep 08 16:17:09.606315 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.606283    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/707a51d2-c191-4bf0-8444-baa02cc338e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-mmjbq\" (UID: \"707a51d2-c191-4bf0-8444-baa02cc338e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"
Sep 08 16:17:09.608828 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.608809    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/707a51d2-c191-4bf0-8444-baa02cc338e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-mmjbq\" (UID: \"707a51d2-c191-4bf0-8444-baa02cc338e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"
Sep 08 16:17:09.700711 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.700608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"
Sep 08 16:17:09.972608 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:09.972514    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"]
Sep 08 16:17:09.975475 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:09.975448    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707a51d2_c191_4bf0_8444_baa02cc338e7.slice/crio-ec1adf54e733a9b5c1140d5bfd85f2ab26ea258afcbc8ecf4122f8e650de6a08 WatchSource:0}: Error finding container ec1adf54e733a9b5c1140d5bfd85f2ab26ea258afcbc8ecf4122f8e650de6a08: Status 404 returned error can't find the container with id ec1adf54e733a9b5c1140d5bfd85f2ab26ea258afcbc8ecf4122f8e650de6a08
Sep 08 16:17:10.341384 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:10.341355    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq" event={"ID":"707a51d2-c191-4bf0-8444-baa02cc338e7","Type":"ContainerStarted","Data":"ec1adf54e733a9b5c1140d5bfd85f2ab26ea258afcbc8ecf4122f8e650de6a08"}
Sep 08 16:17:11.346382 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:11.346341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq" event={"ID":"707a51d2-c191-4bf0-8444-baa02cc338e7","Type":"ContainerStarted","Data":"19c8b79d5b73db673a35e88c7c694b2604404b794b25a0cbde3cbcf3ec06513d"}
Sep 08 16:17:11.346840 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:11.346537    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"
Sep 08 16:17:11.351192 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:11.351167    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq"
Sep 08 16:17:11.362915 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:11.362875    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-mmjbq" podStartSLOduration=1.323919128 podStartE2EDuration="2.36286168s" totalImagesPullingTime="1.038942552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:11.361855152 +0000 UTC m=+265.448518903" watchObservedRunningTime="2026-09-08 16:17:11.36286168 +0000 UTC m=+265.449525427"
Sep 08 16:17:11.522511 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:11.522467    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:17:11.524809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:11.524776    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ede55c97-e9a4-4599-95c5-9b35ca349745-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rjvfx\" (UID: \"ede55c97-e9a4-4599-95c5-9b35ca349745\") " pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:17:11.824940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:11.824914    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mlgpx\""
Sep 08 16:17:11.832925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:11.832906    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rjvfx"
Sep 08 16:17:12.099679 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.099605    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rjvfx"]
Sep 08 16:17:12.102721 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:12.102681    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podede55c97_e9a4_4599_95c5_9b35ca349745.slice/crio-29522ca2dafb4a07d8c427ce1e8a085a9b90b0a203a76ff4eaa069b9cd0e0a5b WatchSource:0}: Error finding container 29522ca2dafb4a07d8c427ce1e8a085a9b90b0a203a76ff4eaa069b9cd0e0a5b: Status 404 returned error can't find the container with id 29522ca2dafb4a07d8c427ce1e8a085a9b90b0a203a76ff4eaa069b9cd0e0a5b
Sep 08 16:17:12.351112 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.351024    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjvfx" event={"ID":"ede55c97-e9a4-4599-95c5-9b35ca349745","Type":"ContainerStarted","Data":"f2818c0d6c3e9acac382ade168313a72e067580bf97ae442c4953e2d2c337a56"}
Sep 08 16:17:12.351112 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.351084    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjvfx" event={"ID":"ede55c97-e9a4-4599-95c5-9b35ca349745","Type":"ContainerStarted","Data":"29522ca2dafb4a07d8c427ce1e8a085a9b90b0a203a76ff4eaa069b9cd0e0a5b"}
Sep 08 16:17:12.447839 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.447798    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-57bbw"]
Sep 08 16:17:12.451011 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.450988    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.454901 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.454866    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 08 16:17:12.454901 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.454888    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-z6qgl\""
Sep 08 16:17:12.455267 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.454888    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 08 16:17:12.455267 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.454981    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 08 16:17:12.455267 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.454892    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 08 16:17:12.455267 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.454890    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 08 16:17:12.459809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.459787    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-57bbw"]
Sep 08 16:17:12.631406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.631315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/520dadb4-b570-48b6-95b4-ef0f455fdfdc-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.631406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.631378    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/520dadb4-b570-48b6-95b4-ef0f455fdfdc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.631627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.631465    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/520dadb4-b570-48b6-95b4-ef0f455fdfdc-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.631627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.631486    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ctcx\" (UniqueName: \"kubernetes.io/projected/520dadb4-b570-48b6-95b4-ef0f455fdfdc-kube-api-access-5ctcx\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.732169 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.732130    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/520dadb4-b570-48b6-95b4-ef0f455fdfdc-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.732347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.732196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/520dadb4-b570-48b6-95b4-ef0f455fdfdc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.732347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.732285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/520dadb4-b570-48b6-95b4-ef0f455fdfdc-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.732347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.732313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ctcx\" (UniqueName: \"kubernetes.io/projected/520dadb4-b570-48b6-95b4-ef0f455fdfdc-kube-api-access-5ctcx\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.732804 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.732776    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/520dadb4-b570-48b6-95b4-ef0f455fdfdc-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.734643 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.734618    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/520dadb4-b570-48b6-95b4-ef0f455fdfdc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.734749 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.734648    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/520dadb4-b570-48b6-95b4-ef0f455fdfdc-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.740027 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.740003    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ctcx\" (UniqueName: \"kubernetes.io/projected/520dadb4-b570-48b6-95b4-ef0f455fdfdc-kube-api-access-5ctcx\") pod \"prometheus-operator-6f4946db56-57bbw\" (UID: \"520dadb4-b570-48b6-95b4-ef0f455fdfdc\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:12.760819 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:12.760790    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw"
Sep 08 16:17:13.066929 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:13.066877    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-57bbw"]
Sep 08 16:17:13.069431 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:13.069396    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520dadb4_b570_48b6_95b4_ef0f455fdfdc.slice/crio-98a484526707dc1fc1f2c0c07cb6f051effd54c951ce5bbb2565c8fa83a60622 WatchSource:0}: Error finding container 98a484526707dc1fc1f2c0c07cb6f051effd54c951ce5bbb2565c8fa83a60622: Status 404 returned error can't find the container with id 98a484526707dc1fc1f2c0c07cb6f051effd54c951ce5bbb2565c8fa83a60622
Sep 08 16:17:13.356417 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:13.356379    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjvfx" event={"ID":"ede55c97-e9a4-4599-95c5-9b35ca349745","Type":"ContainerStarted","Data":"2601aeeb1b5d9ac21d38f503bc140ec310126e6b94f9ce8f2cdc8cd0e35a7bcd"}
Sep 08 16:17:13.357607 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:13.357564    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw" event={"ID":"520dadb4-b570-48b6-95b4-ef0f455fdfdc","Type":"ContainerStarted","Data":"98a484526707dc1fc1f2c0c07cb6f051effd54c951ce5bbb2565c8fa83a60622"}
Sep 08 16:17:15.368043 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:15.368008    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw" event={"ID":"520dadb4-b570-48b6-95b4-ef0f455fdfdc","Type":"ContainerStarted","Data":"f5063bda435d264c37ef7eeccdddf24cfbad9c0ff243ec48b79eac718455cc9a"}
Sep 08 16:17:15.368454 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:15.368069    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw" event={"ID":"520dadb4-b570-48b6-95b4-ef0f455fdfdc","Type":"ContainerStarted","Data":"e79b4c318a80a7a5c73f96f8718fed4ae6c36eb4c513fbb06d98f57033114b18"}
Sep 08 16:17:15.383775 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:15.383728    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-57bbw" podStartSLOduration=1.810824497 podStartE2EDuration="3.383715052s" totalImagesPullingTime="1.572890555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:15.382667941 +0000 UTC m=+269.469331686" watchObservedRunningTime="2026-09-08 16:17:15.383715052 +0000 UTC m=+269.470378799"
Sep 08 16:17:16.374864 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.374835    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rjvfx" event={"ID":"ede55c97-e9a4-4599-95c5-9b35ca349745","Type":"ContainerStarted","Data":"0fd7ca9e9e4b2b23ab8d37cccdfe77cc3a85f3488a8748fb1b36cf3208e8ab19"}
Sep 08 16:17:16.391815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.391772    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rjvfx" podStartSLOduration=33.592623115 podStartE2EDuration="37.391757006s" totalImagesPullingTime="3.799133891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:16:39 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:16.390024595 +0000 UTC m=+270.476688335" watchObservedRunningTime="2026-09-08 16:17:16.391757006 +0000 UTC m=+270.478420769"
Sep 08 16:17:16.847040 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.847000    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b9zx6"]
Sep 08 16:17:16.850455 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.850430    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.853236 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.853183    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 08 16:17:16.853445 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.853424    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 08 16:17:16.853511 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.853447    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 08 16:17:16.853567 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.853543    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2dqlk\""
Sep 08 16:17:16.964185 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zngdw\" (UniqueName: \"kubernetes.io/projected/9a991395-90c0-4fd1-99e5-efa17b433253-kube-api-access-zngdw\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.964185 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964187    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-wtmp\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.964414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-textfile\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.964414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964283    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-root\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.964414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964331    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-tls\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.964414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964364    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a991395-90c0-4fd1-99e5-efa17b433253-metrics-client-ca\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.964414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964381    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-sys\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.964631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:16.964631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:16.964474    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065248 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065213    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a991395-90c0-4fd1-99e5-efa17b433253-metrics-client-ca\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065259    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-sys\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065340    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065403    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zngdw\" (UniqueName: \"kubernetes.io/projected/9a991395-90c0-4fd1-99e5-efa17b433253-kube-api-access-zngdw\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065622 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065430    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-wtmp\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065622 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-sys\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065622 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065465    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-textfile\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065854 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065833    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-textfile\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.065974 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a991395-90c0-4fd1-99e5-efa17b433253-metrics-client-ca\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.066033 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.065983    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-accelerators-collector-config\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.066033 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.066026    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-root\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.066121 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.066068    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-tls\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.066283 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.066264    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-wtmp\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.066337 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.066279    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a991395-90c0-4fd1-99e5-efa17b433253-root\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.068172 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.068146    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.068293 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.068273    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a991395-90c0-4fd1-99e5-efa17b433253-node-exporter-tls\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.074835 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.074812    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zngdw\" (UniqueName: \"kubernetes.io/projected/9a991395-90c0-4fd1-99e5-efa17b433253-kube-api-access-zngdw\") pod \"node-exporter-b9zx6\" (UID: \"9a991395-90c0-4fd1-99e5-efa17b433253\") " pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.163627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.163527    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b9zx6"
Sep 08 16:17:17.175234 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:17.175192    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a991395_90c0_4fd1_99e5_efa17b433253.slice/crio-69bcbf92e004dc6b877c27c52c8e5e33cdcab7b2f17eb27a08728c4c296d101c WatchSource:0}: Error finding container 69bcbf92e004dc6b877c27c52c8e5e33cdcab7b2f17eb27a08728c4c296d101c: Status 404 returned error can't find the container with id 69bcbf92e004dc6b877c27c52c8e5e33cdcab7b2f17eb27a08728c4c296d101c
Sep 08 16:17:17.379647 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:17.379607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9zx6" event={"ID":"9a991395-90c0-4fd1-99e5-efa17b433253","Type":"ContainerStarted","Data":"69bcbf92e004dc6b877c27c52c8e5e33cdcab7b2f17eb27a08728c4c296d101c"}
Sep 08 16:17:18.384872 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:18.384839    2575 generic.go:356] "Generic (PLEG): container finished" podID="9a991395-90c0-4fd1-99e5-efa17b433253" containerID="d32a10ccbf797e83043b2b26efc5d46450e118f2bc9ee8179bdb9a4f1d8cbb25" exitCode=0
Sep 08 16:17:18.385174 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:18.384890    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9zx6" event={"ID":"9a991395-90c0-4fd1-99e5-efa17b433253","Type":"ContainerDied","Data":"d32a10ccbf797e83043b2b26efc5d46450e118f2bc9ee8179bdb9a4f1d8cbb25"}
Sep 08 16:17:19.390452 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:19.390415    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9zx6" event={"ID":"9a991395-90c0-4fd1-99e5-efa17b433253","Type":"ContainerStarted","Data":"d994e6c8e3f1ca020fde225e8538458f58fea9ef0b380a2e3c7e54cf3e668922"}
Sep 08 16:17:19.390815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:19.390477    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b9zx6" event={"ID":"9a991395-90c0-4fd1-99e5-efa17b433253","Type":"ContainerStarted","Data":"3297b53044a4671b4b0f7f61d078eec01ca48521e07db4c4052195b8ee90ae7a"}
Sep 08 16:17:19.411900 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:19.411855    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b9zx6" podStartSLOduration=2.368060944 podStartE2EDuration="3.411841521s" totalImagesPullingTime="981.852003ms" totalInitContainerRuntime="61.928574ms" isStatefulPod=true podCreationTimestamp="2026-09-08 16:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:19.411045392 +0000 UTC m=+273.497709143" watchObservedRunningTime="2026-09-08 16:17:19.411841521 +0000 UTC m=+273.498505268"
Sep 08 16:17:19.500959 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:19.500936    2575 scope.go:117] "RemoveContainer" containerID="1d43a1ee9dbaf51fbd1fa48b4cc6cfaa9c3965780d2488a12772bc1d591859a5"
Sep 08 16:17:20.396042 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.396012    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:17:20.396499 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.396135    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-72szv" event={"ID":"a4fce580-bb38-47e7-880f-90cafa36b3ec","Type":"ContainerStarted","Data":"74ec9b18b0bf26d42192185b45bb4060126646938af80cb2c92df922921b206d"}
Sep 08 16:17:20.405496 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.405469    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-zdfj2"]
Sep 08 16:17:20.409611 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.409590    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-zdfj2"
Sep 08 16:17:20.412649 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.412632    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fjtc8\""
Sep 08 16:17:20.412902 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.412886    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 08 16:17:20.413184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.413171    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 08 16:17:20.422534 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.422513    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-zdfj2"]
Sep 08 16:17:20.423005 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.422971    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-72szv" podStartSLOduration=43.96318692 podStartE2EDuration="46.422958992s" totalImagesPullingTime="2.459772072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:20.421668002 +0000 UTC m=+274.508331752" watchObservedRunningTime="2026-09-08 16:17:20.422958992 +0000 UTC m=+274.509622739"
Sep 08 16:17:20.491978 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.491943    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7lgw\" (UniqueName: \"kubernetes.io/projected/4b11db77-67a6-4083-8c4f-d701e15b92a8-kube-api-access-f7lgw\") pod \"downloads-85d98c998f-zdfj2\" (UID: \"4b11db77-67a6-4083-8c4f-d701e15b92a8\") " pod="openshift-console/downloads-85d98c998f-zdfj2"
Sep 08 16:17:20.593523 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.593487    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7lgw\" (UniqueName: \"kubernetes.io/projected/4b11db77-67a6-4083-8c4f-d701e15b92a8-kube-api-access-f7lgw\") pod \"downloads-85d98c998f-zdfj2\" (UID: \"4b11db77-67a6-4083-8c4f-d701e15b92a8\") " pod="openshift-console/downloads-85d98c998f-zdfj2"
Sep 08 16:17:20.601879 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.601852    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7lgw\" (UniqueName: \"kubernetes.io/projected/4b11db77-67a6-4083-8c4f-d701e15b92a8-kube-api-access-f7lgw\") pod \"downloads-85d98c998f-zdfj2\" (UID: \"4b11db77-67a6-4083-8c4f-d701e15b92a8\") " pod="openshift-console/downloads-85d98c998f-zdfj2"
Sep 08 16:17:20.718071 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.717991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-zdfj2"
Sep 08 16:17:20.998691 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:20.998609    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-zdfj2"]
Sep 08 16:17:21.004170 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:21.004141    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b11db77_67a6_4083_8c4f_d701e15b92a8.slice/crio-f8ab7d7a38af0c9fc60dbc5a6bd5349a213daffd6d982f068a0127b040a28f64 WatchSource:0}: Error finding container f8ab7d7a38af0c9fc60dbc5a6bd5349a213daffd6d982f068a0127b040a28f64: Status 404 returned error can't find the container with id f8ab7d7a38af0c9fc60dbc5a6bd5349a213daffd6d982f068a0127b040a28f64
Sep 08 16:17:21.128020 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.127985    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5b847f7b9-x4g9f"]
Sep 08 16:17:21.132347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.132328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.134889 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.134863    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 08 16:17:21.135020 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.134977    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nm47h\""
Sep 08 16:17:21.135080 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.135024    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 08 16:17:21.135080 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.134977    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 08 16:17:21.135814 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.135793    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 08 16:17:21.135896 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.135870    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-41udj0gqv5hn3\""
Sep 08 16:17:21.142041 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.142017    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b847f7b9-x4g9f"]
Sep 08 16:17:21.198375 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.198346    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6hr5\" (UniqueName: \"kubernetes.io/projected/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-kube-api-access-w6hr5\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.198375 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.198382    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-audit-log\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.198648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.198401    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-secret-metrics-server-client-certs\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.198648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.198437    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-metrics-server-audit-profiles\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.198648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.198508    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.198648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.198571    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-secret-metrics-server-tls\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.198648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.198605    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-client-ca-bundle\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.299527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.299442    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-secret-metrics-server-tls\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.299527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.299476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-client-ca-bundle\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.299527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.299501    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6hr5\" (UniqueName: \"kubernetes.io/projected/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-kube-api-access-w6hr5\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.299527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.299526    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-audit-log\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.299858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.299552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-secret-metrics-server-client-certs\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.299858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.299622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-metrics-server-audit-profiles\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.299858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.299666    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.300021 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.299986    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-audit-log\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.300530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.300499    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.300670 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.300513    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-metrics-server-audit-profiles\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.302105 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.302078    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-secret-metrics-server-client-certs\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.302205 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.302116    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-client-ca-bundle\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.302260 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.302213    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-secret-metrics-server-tls\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.306956 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.306935    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6hr5\" (UniqueName: \"kubernetes.io/projected/f4317f0d-665a-47ac-a7ea-fe020a0a0b40-kube-api-access-w6hr5\") pod \"metrics-server-5b847f7b9-x4g9f\" (UID: \"f4317f0d-665a-47ac-a7ea-fe020a0a0b40\") " pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.400439 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.400404    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-zdfj2" event={"ID":"4b11db77-67a6-4083-8c4f-d701e15b92a8","Type":"ContainerStarted","Data":"f8ab7d7a38af0c9fc60dbc5a6bd5349a213daffd6d982f068a0127b040a28f64"}
Sep 08 16:17:21.442654 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.442621    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:21.572717 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.572693    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"]
Sep 08 16:17:21.576773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.576755    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:21.578958 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.578940    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 08 16:17:21.579053 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.579032    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gkn68\""
Sep 08 16:17:21.582530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.582511    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"]
Sep 08 16:17:21.601682 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.601657    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29221678-084d-4c62-98e3-2fe06530c52f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rv4f6\" (UID: \"29221678-084d-4c62-98e3-2fe06530c52f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:21.601796 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.601707    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw4d7\" (UniqueName: \"kubernetes.io/projected/29221678-084d-4c62-98e3-2fe06530c52f-kube-api-access-mw4d7\") pod \"monitoring-plugin-85bbbbbf4-rv4f6\" (UID: \"29221678-084d-4c62-98e3-2fe06530c52f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:21.703060 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.703027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw4d7\" (UniqueName: \"kubernetes.io/projected/29221678-084d-4c62-98e3-2fe06530c52f-kube-api-access-mw4d7\") pod \"monitoring-plugin-85bbbbbf4-rv4f6\" (UID: \"29221678-084d-4c62-98e3-2fe06530c52f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:21.703216 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.703181    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29221678-084d-4c62-98e3-2fe06530c52f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rv4f6\" (UID: \"29221678-084d-4c62-98e3-2fe06530c52f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:21.703300 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:17:21.703286    2575 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 08 16:17:21.703357 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:17:21.703339    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29221678-084d-4c62-98e3-2fe06530c52f-monitoring-plugin-cert podName:29221678-084d-4c62-98e3-2fe06530c52f nodeName:}" failed. No retries permitted until 2026-09-08 16:17:22.203324149 +0000 UTC m=+276.289987875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/29221678-084d-4c62-98e3-2fe06530c52f-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-rv4f6" (UID: "29221678-084d-4c62-98e3-2fe06530c52f") : secret "monitoring-plugin-cert" not found
Sep 08 16:17:21.710280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.710253    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5b847f7b9-x4g9f"]
Sep 08 16:17:21.713989 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:21.713963    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4317f0d_665a_47ac_a7ea_fe020a0a0b40.slice/crio-e500e44acec74aaa9d7443a9b62122aa01c1a02d43b479b0262811f57280c125 WatchSource:0}: Error finding container e500e44acec74aaa9d7443a9b62122aa01c1a02d43b479b0262811f57280c125: Status 404 returned error can't find the container with id e500e44acec74aaa9d7443a9b62122aa01c1a02d43b479b0262811f57280c125
Sep 08 16:17:21.715355 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:21.715334    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw4d7\" (UniqueName: \"kubernetes.io/projected/29221678-084d-4c62-98e3-2fe06530c52f-kube-api-access-mw4d7\") pod \"monitoring-plugin-85bbbbbf4-rv4f6\" (UID: \"29221678-084d-4c62-98e3-2fe06530c52f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:22.207185 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:22.207148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29221678-084d-4c62-98e3-2fe06530c52f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rv4f6\" (UID: \"29221678-084d-4c62-98e3-2fe06530c52f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:22.209470 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:22.209450    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/29221678-084d-4c62-98e3-2fe06530c52f-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-rv4f6\" (UID: \"29221678-084d-4c62-98e3-2fe06530c52f\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:22.406035 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:22.405995    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f" event={"ID":"f4317f0d-665a-47ac-a7ea-fe020a0a0b40","Type":"ContainerStarted","Data":"e500e44acec74aaa9d7443a9b62122aa01c1a02d43b479b0262811f57280c125"}
Sep 08 16:17:22.487765 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:22.487694    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:22.770740 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:22.770596    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"]
Sep 08 16:17:23.413014 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:23.412973    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6" event={"ID":"29221678-084d-4c62-98e3-2fe06530c52f","Type":"ContainerStarted","Data":"39adb80a37dbd17156cb4adfb41cf0e0b06c1945ac16071662667a30ff3f66a5"}
Sep 08 16:17:24.419235 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:24.419194    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f" event={"ID":"f4317f0d-665a-47ac-a7ea-fe020a0a0b40","Type":"ContainerStarted","Data":"ea0ad5cb6757b00a3c32de548b85061354c9ccea766819cedde51ea4e825696a"}
Sep 08 16:17:24.438459 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:24.438362    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f" podStartSLOduration=1.673795264 podStartE2EDuration="3.438337899s" totalImagesPullingTime="1.764542635s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:24.435505933 +0000 UTC m=+278.522169682" watchObservedRunningTime="2026-09-08 16:17:24.438337899 +0000 UTC m=+278.525001648"
Sep 08 16:17:25.424207 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:25.424174    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6" event={"ID":"29221678-084d-4c62-98e3-2fe06530c52f","Type":"ContainerStarted","Data":"40fa760f7e8b724f4ec2bffa46ab08f9fae61f287ed807db65105694b68ca834"}
Sep 08 16:17:25.424563 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:25.424327    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:25.429018 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:25.428993    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6"
Sep 08 16:17:25.439814 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:25.439778    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-rv4f6" podStartSLOduration=2.474570427 podStartE2EDuration="4.439766576s" totalImagesPullingTime="1.965196149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:25.438942589 +0000 UTC m=+279.525606334" watchObservedRunningTime="2026-09-08 16:17:25.439766576 +0000 UTC m=+279.526430318"
Sep 08 16:17:27.020669 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:17:27.020624    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" podUID="62641f8d-0486-444e-b887-a345668b437f"
Sep 08 16:17:27.020669 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:17:27.020624    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-sl68l" podUID="ed691713-c786-4c14-b382-8bdedeb4073d"
Sep 08 16:17:27.430739 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:27.430708    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:17:27.430739 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:27.430708    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sl68l"
Sep 08 16:17:29.320650 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.320607    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-545dff5cf4-7brwj"]
Sep 08 16:17:29.324973 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.324958    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.327543 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.327515    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 16:17:29.327696 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.327564    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 16:17:29.328931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.328695    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 16:17:29.328931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.328720    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 16:17:29.328931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.328720    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 16:17:29.328931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.328806    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dz7wm\""
Sep 08 16:17:29.331860 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.331840    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-545dff5cf4-7brwj"]
Sep 08 16:17:29.376424 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.376395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-config\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.376544 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.376434    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-oauth-serving-cert\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.376544 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.376491    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chkv9\" (UniqueName: \"kubernetes.io/projected/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-kube-api-access-chkv9\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.376544 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.376530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-oauth-config\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.376681 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.376618    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-serving-cert\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.376681 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.376654    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-service-ca\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.477126 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-config\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.477239 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477131    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-oauth-serving-cert\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.477239 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477171    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chkv9\" (UniqueName: \"kubernetes.io/projected/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-kube-api-access-chkv9\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.477239 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477212    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-oauth-config\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.477391 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477242    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-serving-cert\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.477391 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477277    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-service-ca\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.477904 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477881    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-config\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.478024 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477999    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-service-ca\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.478085 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.477999    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-oauth-serving-cert\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.479523 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.479505    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-oauth-config\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.479702 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.479687    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-serving-cert\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.484461 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.484435    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chkv9\" (UniqueName: \"kubernetes.io/projected/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-kube-api-access-chkv9\") pod \"console-545dff5cf4-7brwj\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") " pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.636099 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.636069    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:29.905619 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:29.905525    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-545dff5cf4-7brwj"]
Sep 08 16:17:29.909131 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:29.909107    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8cb1800_1b6e_4ad9_958f_5a7a3e0f5f04.slice/crio-2cd13fbf644d3d69b91cbc80c8dcbfbe3936134a284d1fdb8820c17274ea6da2 WatchSource:0}: Error finding container 2cd13fbf644d3d69b91cbc80c8dcbfbe3936134a284d1fdb8820c17274ea6da2: Status 404 returned error can't find the container with id 2cd13fbf644d3d69b91cbc80c8dcbfbe3936134a284d1fdb8820c17274ea6da2
Sep 08 16:17:30.182752 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.182672    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:17:30.182752 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.182706    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:17:30.184985 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.184965    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ed691713-c786-4c14-b382-8bdedeb4073d-metrics-tls\") pod \"dns-default-sl68l\" (UID: \"ed691713-c786-4c14-b382-8bdedeb4073d\") " pod="openshift-dns/dns-default-sl68l"
Sep 08 16:17:30.185059 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.184997    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"image-registry-6c595d89b8-mxfw5\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") " pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:17:30.396663 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.396625    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:17:30.400303 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.400281    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-72szv"
Sep 08 16:17:30.434567 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.434500    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgmfx\""
Sep 08 16:17:30.434567 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.434542    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nlhzx\""
Sep 08 16:17:30.442168 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.442147    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sl68l"
Sep 08 16:17:30.442288 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.442266    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:17:30.442921 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.442893    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545dff5cf4-7brwj" event={"ID":"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04","Type":"ContainerStarted","Data":"2cd13fbf644d3d69b91cbc80c8dcbfbe3936134a284d1fdb8820c17274ea6da2"}
Sep 08 16:17:30.722075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.722040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sl68l"]
Sep 08 16:17:30.724043 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:30.724020    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c595d89b8-mxfw5"]
Sep 08 16:17:30.724319 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:30.724296    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded691713_c786_4c14_b382_8bdedeb4073d.slice/crio-2e7a8ed181466508797cc17519976c8665d2ad7b01b549bf82cb243f49996eca WatchSource:0}: Error finding container 2e7a8ed181466508797cc17519976c8665d2ad7b01b549bf82cb243f49996eca: Status 404 returned error can't find the container with id 2e7a8ed181466508797cc17519976c8665d2ad7b01b549bf82cb243f49996eca
Sep 08 16:17:30.726850 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:17:30.726826    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62641f8d_0486_444e_b887_a345668b437f.slice/crio-b406d9321998a97e0d020bdbab7a5b1ade36e585b353822c608e9ae4d2187043 WatchSource:0}: Error finding container b406d9321998a97e0d020bdbab7a5b1ade36e585b353822c608e9ae4d2187043: Status 404 returned error can't find the container with id b406d9321998a97e0d020bdbab7a5b1ade36e585b353822c608e9ae4d2187043
Sep 08 16:17:31.449063 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:31.448979    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sl68l" event={"ID":"ed691713-c786-4c14-b382-8bdedeb4073d","Type":"ContainerStarted","Data":"2e7a8ed181466508797cc17519976c8665d2ad7b01b549bf82cb243f49996eca"}
Sep 08 16:17:31.451238 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:31.451188    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" event={"ID":"62641f8d-0486-444e-b887-a345668b437f","Type":"ContainerStarted","Data":"38a8f99e8a01639b1c835ff014028bff95a82e7a78af5d71275f2bdffd7b62a0"}
Sep 08 16:17:31.451365 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:31.451261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" event={"ID":"62641f8d-0486-444e-b887-a345668b437f","Type":"ContainerStarted","Data":"b406d9321998a97e0d020bdbab7a5b1ade36e585b353822c608e9ae4d2187043"}
Sep 08 16:17:31.451846 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:31.451811    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:17:31.473914 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:31.473240    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" podStartSLOduration=284.473220121 podStartE2EDuration="4m44.473220121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:31.471532916 +0000 UTC m=+285.558196664" watchObservedRunningTime="2026-09-08 16:17:31.473220121 +0000 UTC m=+285.559883869"
Sep 08 16:17:34.468308 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:34.468208    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sl68l" event={"ID":"ed691713-c786-4c14-b382-8bdedeb4073d","Type":"ContainerStarted","Data":"1b85b6d6f7536ecde66f2397f460a95629b8ad7f3458d9c854fba5ab74337726"}
Sep 08 16:17:34.468761 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:34.468421    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sl68l"
Sep 08 16:17:34.468761 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:34.468471    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sl68l" event={"ID":"ed691713-c786-4c14-b382-8bdedeb4073d","Type":"ContainerStarted","Data":"ef4d0e0123ae43ad50f9bafba5030264f6360caf85644de1dcca1b3548b26ff5"}
Sep 08 16:17:34.469968 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:34.469937    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545dff5cf4-7brwj" event={"ID":"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04","Type":"ContainerStarted","Data":"21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c"}
Sep 08 16:17:34.487099 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:34.487044    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sl68l" podStartSLOduration=251.062471749 podStartE2EDuration="4m14.487027365s" totalImagesPullingTime="3.424555616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:13:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:34.485460194 +0000 UTC m=+288.572123943" watchObservedRunningTime="2026-09-08 16:17:34.487027365 +0000 UTC m=+288.573691115"
Sep 08 16:17:34.504388 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:34.504338    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-545dff5cf4-7brwj" podStartSLOduration=1.259839803 podStartE2EDuration="5.504318753s" totalImagesPullingTime="4.24447895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:34.503487424 +0000 UTC m=+288.590151173" watchObservedRunningTime="2026-09-08 16:17:34.504318753 +0000 UTC m=+288.590982503"
Sep 08 16:17:39.636730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:39.636682    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:39.637315 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:39.636767    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:17:39.638325 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:39.638299    2575 patch_prober.go:28] interesting pod/console-545dff5cf4-7brwj container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused" start-of-body=
Sep 08 16:17:39.638438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:39.638398    2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-545dff5cf4-7brwj" podUID="e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" containerName="console" probeResult="failure" output="Get \"https://10.133.0.20:8443/health\": dial tcp 10.133.0.20:8443: connect: connection refused"
Sep 08 16:17:41.442985 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:41.442951    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:41.443386 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:41.443011    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:17:44.475542 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:44.475513    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sl68l"
Sep 08 16:17:45.520637 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:45.520574    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-zdfj2" event={"ID":"4b11db77-67a6-4083-8c4f-d701e15b92a8","Type":"ContainerStarted","Data":"f2d1702387d15cbd0de3dce002a1a4e079b9fd1b33444a6c0c92786506938a54"}
Sep 08 16:17:45.521116 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:45.520696    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-zdfj2"
Sep 08 16:17:45.535017 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:45.534987    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-zdfj2"
Sep 08 16:17:45.537687 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:45.537640    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-zdfj2" podStartSLOduration=2.040975742 podStartE2EDuration="25.537625383s" totalImagesPullingTime="23.496649641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:17:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:17:45.536392477 +0000 UTC m=+299.623056230" watchObservedRunningTime="2026-09-08 16:17:45.537625383 +0000 UTC m=+299.624289140"
Sep 08 16:17:46.274743 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:46.274710    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:17:46.275378 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:46.275352    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:17:46.283734 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:46.283709    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:17:46.284369 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:46.284345    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:17:46.292164 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:46.292109    2575 kubelet.go:1648] "Image garbage collection succeeded"
Sep 08 16:17:48.978763 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:48.978715    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-545dff5cf4-7brwj"]
Sep 08 16:17:50.446709 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:50.446673    2575 patch_prober.go:28] interesting pod/image-registry-6c595d89b8-mxfw5 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 08 16:17:50.447203 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:50.446741    2575 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" podUID="62641f8d-0486-444e-b887-a345668b437f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 08 16:17:52.462018 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:52.461989    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:17:53.554117 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:53.554049    2575 generic.go:356] "Generic (PLEG): container finished" podID="9eb7eb53-77fc-4799-bc28-0f5d15ec2952" containerID="2f697af2b51cbcb5fa0ed1c84d809c95d72985ed5fb8e4c84de3a55bc13f0a8d" exitCode=1
Sep 08 16:17:53.554616 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:53.554119    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" event={"ID":"9eb7eb53-77fc-4799-bc28-0f5d15ec2952","Type":"ContainerDied","Data":"2f697af2b51cbcb5fa0ed1c84d809c95d72985ed5fb8e4c84de3a55bc13f0a8d"}
Sep 08 16:17:53.554616 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:53.554175    2575 scope.go:117] "RemoveContainer" containerID="37db98e81c075388eae11c65b6c293ab03c854494c2d7bee6d2dfea9c3ee34e2"
Sep 08 16:17:53.554616 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:53.554409    2575 scope.go:117] "RemoveContainer" containerID="2f697af2b51cbcb5fa0ed1c84d809c95d72985ed5fb8e4c84de3a55bc13f0a8d"
Sep 08 16:17:53.554735 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:17:53.554619    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-ffccc8756-sjn97_open-cluster-management-agent-addon(9eb7eb53-77fc-4799-bc28-0f5d15ec2952)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" podUID="9eb7eb53-77fc-4799-bc28-0f5d15ec2952"
Sep 08 16:17:54.095298 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:54.095261    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:17:54.558876 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:54.558813    2575 scope.go:117] "RemoveContainer" containerID="2f697af2b51cbcb5fa0ed1c84d809c95d72985ed5fb8e4c84de3a55bc13f0a8d"
Sep 08 16:17:54.559212 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:17:54.558952    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-ffccc8756-sjn97_open-cluster-management-agent-addon(9eb7eb53-77fc-4799-bc28-0f5d15ec2952)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" podUID="9eb7eb53-77fc-4799-bc28-0f5d15ec2952"
Sep 08 16:17:55.915059 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:55.915024    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c595d89b8-mxfw5"]
Sep 08 16:17:57.389457 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:57.389415    2575 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:17:57.389856 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:17:57.389783    2575 scope.go:117] "RemoveContainer" containerID="2f697af2b51cbcb5fa0ed1c84d809c95d72985ed5fb8e4c84de3a55bc13f0a8d"
Sep 08 16:17:57.389946 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:17:57.389924    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"acm-agent\" with CrashLoopBackOff: \"back-off 10s restarting failed container=acm-agent pod=klusterlet-addon-workmgr-ffccc8756-sjn97_open-cluster-management-agent-addon(9eb7eb53-77fc-4799-bc28-0f5d15ec2952)\"" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" podUID="9eb7eb53-77fc-4799-bc28-0f5d15ec2952"
Sep 08 16:18:01.448193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:01.448162    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:18:01.451952 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:01.451931    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5b847f7b9-x4g9f"
Sep 08 16:18:09.995235 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:09.995209    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-849b99ddd-v5nhz_28efe48a-f884-4d00-a8dd-f33b1a14bae4/router/0.log"
Sep 08 16:18:10.006384 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:10.006363    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fnfbq_d4599713-5ceb-4e41-a84e-0256209180b7/serve-healthcheck-canary/0.log"
Sep 08 16:18:11.500944 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:11.500919    2575 scope.go:117] "RemoveContainer" containerID="2f697af2b51cbcb5fa0ed1c84d809c95d72985ed5fb8e4c84de3a55bc13f0a8d"
Sep 08 16:18:11.617119 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:11.617093    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97" event={"ID":"9eb7eb53-77fc-4799-bc28-0f5d15ec2952","Type":"ContainerStarted","Data":"0ad33340231e8db65b58ca448a68e97bd9af3c0d4df5b7a5d090590447d3e02b"}
Sep 08 16:18:11.617451 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:11.617424    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:18:11.618642 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:11.618622    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-ffccc8756-sjn97"
Sep 08 16:18:14.002596 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.002537    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-545dff5cf4-7brwj" podUID="e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" containerName="console" containerID="cri-o://21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c" gracePeriod=15
Sep 08 16:18:14.268529 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.268476    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:18:14.433255 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433225    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-oauth-config\") pod \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") "
Sep 08 16:18:14.433409 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433264    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chkv9\" (UniqueName: \"kubernetes.io/projected/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-kube-api-access-chkv9\") pod \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") "
Sep 08 16:18:14.433409 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433288    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-service-ca\") pod \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") "
Sep 08 16:18:14.433409 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433349    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-oauth-serving-cert\") pod \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") "
Sep 08 16:18:14.433409 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433380    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-config\") pod \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") "
Sep 08 16:18:14.433624 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433414    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-serving-cert\") pod \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\" (UID: \"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04\") "
Sep 08 16:18:14.433774 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433745    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-service-ca" (OuterVolumeSpecName: "service-ca") pod "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" (UID: "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:14.433846 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433769    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" (UID: "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:14.433846 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.433774    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-config" (OuterVolumeSpecName: "console-config") pod "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" (UID: "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:14.435471 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.435443    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" (UID: "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:14.435559 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.435454    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" (UID: "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:14.435559 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.435454    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-kube-api-access-chkv9" (OuterVolumeSpecName: "kube-api-access-chkv9") pod "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" (UID: "e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04"). InnerVolumeSpecName "kube-api-access-chkv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:18:14.534537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.534491    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:14.534537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.534509    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-oauth-serving-cert\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:14.534537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.534518    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-config\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:14.534537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.534527    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-serving-cert\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:14.534537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.534535    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-console-oauth-config\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:14.534767 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.534543    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chkv9\" (UniqueName: \"kubernetes.io/projected/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04-kube-api-access-chkv9\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:14.629470 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.629442    2575 generic.go:356] "Generic (PLEG): container finished" podID="e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" containerID="21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c" exitCode=0
Sep 08 16:18:14.629555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.629487    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-545dff5cf4-7brwj"
Sep 08 16:18:14.629555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.629525    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545dff5cf4-7brwj" event={"ID":"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04","Type":"ContainerDied","Data":"21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c"}
Sep 08 16:18:14.629664 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.629600    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-545dff5cf4-7brwj" event={"ID":"e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04","Type":"ContainerDied","Data":"2cd13fbf644d3d69b91cbc80c8dcbfbe3936134a284d1fdb8820c17274ea6da2"}
Sep 08 16:18:14.629664 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.629629    2575 scope.go:117] "RemoveContainer" containerID="21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c"
Sep 08 16:18:14.641211 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.641194    2575 scope.go:117] "RemoveContainer" containerID="21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c"
Sep 08 16:18:14.641463 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:18:14.641442    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c\": container with ID starting with 21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c not found: ID does not exist" containerID="21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c"
Sep 08 16:18:14.641513 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.641470    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c"} err="failed to get container status \"21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c\": rpc error: code = NotFound desc = could not find container \"21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c\": container with ID starting with 21e22ede02911ceb8d6f51796009a8cb9361dc1a809d8b5de97adaaddd02002c not found: ID does not exist"
Sep 08 16:18:14.645343 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.645325    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-545dff5cf4-7brwj"]
Sep 08 16:18:14.648977 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:14.648957    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-545dff5cf4-7brwj"]
Sep 08 16:18:16.505317 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:16.505285    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" path="/var/lib/kubelet/pods/e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04/volumes"
Sep 08 16:18:20.941013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:20.940953    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" podUID="62641f8d-0486-444e-b887-a345668b437f" containerName="registry" containerID="cri-o://38a8f99e8a01639b1c835ff014028bff95a82e7a78af5d71275f2bdffd7b62a0" gracePeriod=30
Sep 08 16:18:22.458189 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.458157    2575 patch_prober.go:28] interesting pod/image-registry-6c595d89b8-mxfw5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.133.0.6:5000/healthz\": dial tcp 10.133.0.6:5000: connect: connection refused" start-of-body=
Sep 08 16:18:22.458636 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.458213    2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" podUID="62641f8d-0486-444e-b887-a345668b437f" containerName="registry" probeResult="failure" output="Get \"https://10.133.0.6:5000/healthz\": dial tcp 10.133.0.6:5000: connect: connection refused"
Sep 08 16:18:22.676306 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.676273    2575 generic.go:356] "Generic (PLEG): container finished" podID="62641f8d-0486-444e-b887-a345668b437f" containerID="38a8f99e8a01639b1c835ff014028bff95a82e7a78af5d71275f2bdffd7b62a0" exitCode=0
Sep 08 16:18:22.676444 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.676346    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" event={"ID":"62641f8d-0486-444e-b887-a345668b437f","Type":"ContainerDied","Data":"38a8f99e8a01639b1c835ff014028bff95a82e7a78af5d71275f2bdffd7b62a0"}
Sep 08 16:18:22.770097 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.769756    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:18:22.791205 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.791174    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-image-registry-private-configuration\") pod \"62641f8d-0486-444e-b887-a345668b437f\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") "
Sep 08 16:18:22.791381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.791243    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62641f8d-0486-444e-b887-a345668b437f-ca-trust-extracted\") pod \"62641f8d-0486-444e-b887-a345668b437f\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") "
Sep 08 16:18:22.791381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.791271    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-installation-pull-secrets\") pod \"62641f8d-0486-444e-b887-a345668b437f\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") "
Sep 08 16:18:22.791381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.791313    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-bound-sa-token\") pod \"62641f8d-0486-444e-b887-a345668b437f\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") "
Sep 08 16:18:22.791381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.791338    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") pod \"62641f8d-0486-444e-b887-a345668b437f\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") "
Sep 08 16:18:22.791381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.791371    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-registry-certificates\") pod \"62641f8d-0486-444e-b887-a345668b437f\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") "
Sep 08 16:18:22.791381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.791397    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6bxv\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-kube-api-access-z6bxv\") pod \"62641f8d-0486-444e-b887-a345668b437f\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") "
Sep 08 16:18:22.791795 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.791493    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-trusted-ca\") pod \"62641f8d-0486-444e-b887-a345668b437f\" (UID: \"62641f8d-0486-444e-b887-a345668b437f\") "
Sep 08 16:18:22.793176 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.793123    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "62641f8d-0486-444e-b887-a345668b437f" (UID: "62641f8d-0486-444e-b887-a345668b437f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:22.794471 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.794429    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "62641f8d-0486-444e-b887-a345668b437f" (UID: "62641f8d-0486-444e-b887-a345668b437f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:18:22.795895 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.795481    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "62641f8d-0486-444e-b887-a345668b437f" (UID: "62641f8d-0486-444e-b887-a345668b437f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:22.796063 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.796030    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "62641f8d-0486-444e-b887-a345668b437f" (UID: "62641f8d-0486-444e-b887-a345668b437f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:18:22.796328 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.796290    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "62641f8d-0486-444e-b887-a345668b437f" (UID: "62641f8d-0486-444e-b887-a345668b437f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:18:22.796962 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.796383    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "62641f8d-0486-444e-b887-a345668b437f" (UID: "62641f8d-0486-444e-b887-a345668b437f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:18:22.797849 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.797603    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-kube-api-access-z6bxv" (OuterVolumeSpecName: "kube-api-access-z6bxv") pod "62641f8d-0486-444e-b887-a345668b437f" (UID: "62641f8d-0486-444e-b887-a345668b437f"). InnerVolumeSpecName "kube-api-access-z6bxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:18:22.804685 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.804657    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62641f8d-0486-444e-b887-a345668b437f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "62641f8d-0486-444e-b887-a345668b437f" (UID: "62641f8d-0486-444e-b887-a345668b437f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:18:22.893132 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.893092    2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-image-registry-private-configuration\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:22.893272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.893138    2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/62641f8d-0486-444e-b887-a345668b437f-ca-trust-extracted\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:22.893272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.893156    2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/62641f8d-0486-444e-b887-a345668b437f-installation-pull-secrets\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:22.893272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.893173    2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-bound-sa-token\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:22.893272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.893189    2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-registry-tls\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:22.893272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.893203    2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-registry-certificates\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:22.893272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.893219    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6bxv\" (UniqueName: \"kubernetes.io/projected/62641f8d-0486-444e-b887-a345668b437f-kube-api-access-z6bxv\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:22.893272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:22.893233    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62641f8d-0486-444e-b887-a345668b437f-trusted-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:18:23.682508 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:23.682485    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5"
Sep 08 16:18:23.682508 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:23.682505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c595d89b8-mxfw5" event={"ID":"62641f8d-0486-444e-b887-a345668b437f","Type":"ContainerDied","Data":"b406d9321998a97e0d020bdbab7a5b1ade36e585b353822c608e9ae4d2187043"}
Sep 08 16:18:23.682960 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:23.682558    2575 scope.go:117] "RemoveContainer" containerID="38a8f99e8a01639b1c835ff014028bff95a82e7a78af5d71275f2bdffd7b62a0"
Sep 08 16:18:23.702404 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:23.702383    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c595d89b8-mxfw5"]
Sep 08 16:18:23.704689 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:23.704669    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c595d89b8-mxfw5"]
Sep 08 16:18:24.507243 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:24.507208    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62641f8d-0486-444e-b887-a345668b437f" path="/var/lib/kubelet/pods/62641f8d-0486-444e-b887-a345668b437f/volumes"
Sep 08 16:18:44.281506 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.281421    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fccf57cc5-d2gt5"]
Sep 08 16:18:44.281955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.281844    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62641f8d-0486-444e-b887-a345668b437f" containerName="registry"
Sep 08 16:18:44.281955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.281861    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="62641f8d-0486-444e-b887-a345668b437f" containerName="registry"
Sep 08 16:18:44.281955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.281870    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" containerName="console"
Sep 08 16:18:44.281955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.281876    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" containerName="console"
Sep 08 16:18:44.281955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.281948    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="62641f8d-0486-444e-b887-a345668b437f" containerName="registry"
Sep 08 16:18:44.281955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.281957    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8cb1800-1b6e-4ad9-958f-5a7a3e0f5f04" containerName="console"
Sep 08 16:18:44.283999 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.283977    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.289358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.289331    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 16:18:44.289535 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.289513    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 16:18:44.289709 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.289377    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 16:18:44.289827 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.289397    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 16:18:44.289827 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.289442    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 16:18:44.289827 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.289338    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dz7wm\""
Sep 08 16:18:44.293649 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.293632    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 16:18:44.297400 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.297376    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fccf57cc5-d2gt5"]
Sep 08 16:18:44.351679 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.351656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-oauth-serving-cert\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.351784 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.351712    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-oauth-config\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.351824 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.351793    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-service-ca\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.351858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.351826    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-serving-cert\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.351858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.351850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-config\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.351954 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.351884    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5tg2\" (UniqueName: \"kubernetes.io/projected/31eb1fe4-6d51-446c-b42b-6b62393ac293-kube-api-access-t5tg2\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.351954 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.351906    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-trusted-ca-bundle\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.452825 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.452800    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-service-ca\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.452922 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.452829    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-serving-cert\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.452922 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.452849    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-config\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.452922 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.452865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5tg2\" (UniqueName: \"kubernetes.io/projected/31eb1fe4-6d51-446c-b42b-6b62393ac293-kube-api-access-t5tg2\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.452922 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.452884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-trusted-ca-bundle\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.453095 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.453027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-oauth-serving-cert\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.453157 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.453128    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-oauth-config\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.453626 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.453575    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-service-ca\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.453726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.453684    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-oauth-serving-cert\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.453726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.453688    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-config\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.453846 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.453749    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-trusted-ca-bundle\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.455289 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.455269    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-serving-cert\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.455439 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.455422    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-oauth-config\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.461121 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.461100    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5tg2\" (UniqueName: \"kubernetes.io/projected/31eb1fe4-6d51-446c-b42b-6b62393ac293-kube-api-access-t5tg2\") pod \"console-7fccf57cc5-d2gt5\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") " pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.596845 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.596825    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:44.869692 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:44.869617    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fccf57cc5-d2gt5"]
Sep 08 16:18:44.877034 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:18:44.877004    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31eb1fe4_6d51_446c_b42b_6b62393ac293.slice/crio-15b9689f701288df39060f21fa2c14fe4f326e499cad846629d87dbe49e6db61 WatchSource:0}: Error finding container 15b9689f701288df39060f21fa2c14fe4f326e499cad846629d87dbe49e6db61: Status 404 returned error can't find the container with id 15b9689f701288df39060f21fa2c14fe4f326e499cad846629d87dbe49e6db61
Sep 08 16:18:45.768393 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:45.768358    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fccf57cc5-d2gt5" event={"ID":"31eb1fe4-6d51-446c-b42b-6b62393ac293","Type":"ContainerStarted","Data":"f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd"}
Sep 08 16:18:45.768768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:45.768405    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fccf57cc5-d2gt5" event={"ID":"31eb1fe4-6d51-446c-b42b-6b62393ac293","Type":"ContainerStarted","Data":"15b9689f701288df39060f21fa2c14fe4f326e499cad846629d87dbe49e6db61"}
Sep 08 16:18:45.787567 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:45.787523    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fccf57cc5-d2gt5" podStartSLOduration=1.787509097 podStartE2EDuration="1.787509097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:18:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:18:45.78624623 +0000 UTC m=+359.872909978" watchObservedRunningTime="2026-09-08 16:18:45.787509097 +0000 UTC m=+359.874172844"
Sep 08 16:18:54.597750 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:54.597721    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:54.598203 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:54.597769    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:54.602501 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:54.602476    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:18:54.809198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:18:54.809174    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:20:31.648444 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:31.648402    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fccf57cc5-d2gt5"]
Sep 08 16:20:56.674641 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.674560    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fccf57cc5-d2gt5" podUID="31eb1fe4-6d51-446c-b42b-6b62393ac293" containerName="console" containerID="cri-o://f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd" gracePeriod=15
Sep 08 16:20:56.902217 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.902195    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:20:56.959401 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959334    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-oauth-serving-cert\") pod \"31eb1fe4-6d51-446c-b42b-6b62393ac293\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") "
Sep 08 16:20:56.959401 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959367    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-serving-cert\") pod \"31eb1fe4-6d51-446c-b42b-6b62393ac293\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") "
Sep 08 16:20:56.959401 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959390    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-oauth-config\") pod \"31eb1fe4-6d51-446c-b42b-6b62393ac293\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") "
Sep 08 16:20:56.959658 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959406    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-trusted-ca-bundle\") pod \"31eb1fe4-6d51-446c-b42b-6b62393ac293\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") "
Sep 08 16:20:56.959658 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959440    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-service-ca\") pod \"31eb1fe4-6d51-446c-b42b-6b62393ac293\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") "
Sep 08 16:20:56.959658 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959608    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5tg2\" (UniqueName: \"kubernetes.io/projected/31eb1fe4-6d51-446c-b42b-6b62393ac293-kube-api-access-t5tg2\") pod \"31eb1fe4-6d51-446c-b42b-6b62393ac293\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") "
Sep 08 16:20:56.959820 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959665    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-config\") pod \"31eb1fe4-6d51-446c-b42b-6b62393ac293\" (UID: \"31eb1fe4-6d51-446c-b42b-6b62393ac293\") "
Sep 08 16:20:56.959882 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959813    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "31eb1fe4-6d51-446c-b42b-6b62393ac293" (UID: "31eb1fe4-6d51-446c-b42b-6b62393ac293"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:20:56.959882 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959865    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-service-ca" (OuterVolumeSpecName: "service-ca") pod "31eb1fe4-6d51-446c-b42b-6b62393ac293" (UID: "31eb1fe4-6d51-446c-b42b-6b62393ac293"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:20:56.959979 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.959911    2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-oauth-serving-cert\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:20:56.960082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.960049    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "31eb1fe4-6d51-446c-b42b-6b62393ac293" (UID: "31eb1fe4-6d51-446c-b42b-6b62393ac293"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:20:56.960207 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.960100    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-config" (OuterVolumeSpecName: "console-config") pod "31eb1fe4-6d51-446c-b42b-6b62393ac293" (UID: "31eb1fe4-6d51-446c-b42b-6b62393ac293"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:20:56.961543 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.961522    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "31eb1fe4-6d51-446c-b42b-6b62393ac293" (UID: "31eb1fe4-6d51-446c-b42b-6b62393ac293"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:20:56.961638 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.961544    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "31eb1fe4-6d51-446c-b42b-6b62393ac293" (UID: "31eb1fe4-6d51-446c-b42b-6b62393ac293"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:20:56.961638 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:56.961540    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31eb1fe4-6d51-446c-b42b-6b62393ac293-kube-api-access-t5tg2" (OuterVolumeSpecName: "kube-api-access-t5tg2") pod "31eb1fe4-6d51-446c-b42b-6b62393ac293" (UID: "31eb1fe4-6d51-446c-b42b-6b62393ac293"). InnerVolumeSpecName "kube-api-access-t5tg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:20:57.060662 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.060640    2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-config\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:20:57.060662 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.060659    2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-serving-cert\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:20:57.060788 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.060668    2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/31eb1fe4-6d51-446c-b42b-6b62393ac293-console-oauth-config\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:20:57.060788 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.060678    2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-trusted-ca-bundle\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:20:57.060788 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.060686    2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/31eb1fe4-6d51-446c-b42b-6b62393ac293-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:20:57.060788 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.060694    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5tg2\" (UniqueName: \"kubernetes.io/projected/31eb1fe4-6d51-446c-b42b-6b62393ac293-kube-api-access-t5tg2\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:20:57.251347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.251275    2575 generic.go:356] "Generic (PLEG): container finished" podID="31eb1fe4-6d51-446c-b42b-6b62393ac293" containerID="f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd" exitCode=0
Sep 08 16:20:57.251347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.251321    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fccf57cc5-d2gt5"
Sep 08 16:20:57.251469 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.251365    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fccf57cc5-d2gt5" event={"ID":"31eb1fe4-6d51-446c-b42b-6b62393ac293","Type":"ContainerDied","Data":"f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd"}
Sep 08 16:20:57.251469 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.251418    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fccf57cc5-d2gt5" event={"ID":"31eb1fe4-6d51-446c-b42b-6b62393ac293","Type":"ContainerDied","Data":"15b9689f701288df39060f21fa2c14fe4f326e499cad846629d87dbe49e6db61"}
Sep 08 16:20:57.251469 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.251442    2575 scope.go:117] "RemoveContainer" containerID="f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd"
Sep 08 16:20:57.259655 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.259634    2575 scope.go:117] "RemoveContainer" containerID="f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd"
Sep 08 16:20:57.259893 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:20:57.259878    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd\": container with ID starting with f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd not found: ID does not exist" containerID="f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd"
Sep 08 16:20:57.259944 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.259900    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd"} err="failed to get container status \"f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd\": rpc error: code = NotFound desc = could not find container \"f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd\": container with ID starting with f1c31aca7379f6a8516d89612c731e39c09c772477cf7469b705b9a203c35ebd not found: ID does not exist"
Sep 08 16:20:57.270686 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.270668    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fccf57cc5-d2gt5"]
Sep 08 16:20:57.272753 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:57.272736    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fccf57cc5-d2gt5"]
Sep 08 16:20:58.505474 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:20:58.505431    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31eb1fe4-6d51-446c-b42b-6b62393ac293" path="/var/lib/kubelet/pods/31eb1fe4-6d51-446c-b42b-6b62393ac293/volumes"
Sep 08 16:21:43.459925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.459847    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c84lt"]
Sep 08 16:21:43.460379 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.460178    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31eb1fe4-6d51-446c-b42b-6b62393ac293" containerName="console"
Sep 08 16:21:43.460379 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.460192    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="31eb1fe4-6d51-446c-b42b-6b62393ac293" containerName="console"
Sep 08 16:21:43.460379 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.460238    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="31eb1fe4-6d51-446c-b42b-6b62393ac293" containerName="console"
Sep 08 16:21:43.462002 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.461987    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:43.464219 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.464193    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 08 16:21:43.465139 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.465120    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vwpzh\""
Sep 08 16:21:43.465258 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.465168    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 08 16:21:43.469704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.469684    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c84lt"]
Sep 08 16:21:43.579084 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.579045    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/babe2c51-e70e-482d-af23-09e415337533-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c84lt\" (UID: \"babe2c51-e70e-482d-af23-09e415337533\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:43.579253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.579105    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rwqd\" (UniqueName: \"kubernetes.io/projected/babe2c51-e70e-482d-af23-09e415337533-kube-api-access-7rwqd\") pod \"cert-manager-webhook-6486ffd997-c84lt\" (UID: \"babe2c51-e70e-482d-af23-09e415337533\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:43.680157 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.680122    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/babe2c51-e70e-482d-af23-09e415337533-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c84lt\" (UID: \"babe2c51-e70e-482d-af23-09e415337533\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:43.680339 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.680211    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rwqd\" (UniqueName: \"kubernetes.io/projected/babe2c51-e70e-482d-af23-09e415337533-kube-api-access-7rwqd\") pod \"cert-manager-webhook-6486ffd997-c84lt\" (UID: \"babe2c51-e70e-482d-af23-09e415337533\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:43.687756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.687724    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/babe2c51-e70e-482d-af23-09e415337533-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-c84lt\" (UID: \"babe2c51-e70e-482d-af23-09e415337533\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:43.687875 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.687785    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rwqd\" (UniqueName: \"kubernetes.io/projected/babe2c51-e70e-482d-af23-09e415337533-kube-api-access-7rwqd\") pod \"cert-manager-webhook-6486ffd997-c84lt\" (UID: \"babe2c51-e70e-482d-af23-09e415337533\") " pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:43.778571 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:43.778479    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:44.044019 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:44.043934    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-c84lt"]
Sep 08 16:21:44.047008 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:21:44.046973    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbabe2c51_e70e_482d_af23_09e415337533.slice/crio-232e00010f918f9284a392f7d5c14554674558a939285f924946d9aec0aa845f WatchSource:0}: Error finding container 232e00010f918f9284a392f7d5c14554674558a939285f924946d9aec0aa845f: Status 404 returned error can't find the container with id 232e00010f918f9284a392f7d5c14554674558a939285f924946d9aec0aa845f
Sep 08 16:21:44.048891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:44.048873    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:21:44.418013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:44.417974    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt" event={"ID":"babe2c51-e70e-482d-af23-09e415337533","Type":"ContainerStarted","Data":"232e00010f918f9284a392f7d5c14554674558a939285f924946d9aec0aa845f"}
Sep 08 16:21:47.430380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:47.430342    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt" event={"ID":"babe2c51-e70e-482d-af23-09e415337533","Type":"ContainerStarted","Data":"1e8bb12029b01e7ed1e2bbe090a4259dc66700804dd3d88ee143adf6794f5bd3"}
Sep 08 16:21:47.430722 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:47.430536    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:21:47.446337 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:47.446294    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt" podStartSLOduration=1.469161497 podStartE2EDuration="4.446279042s" totalImagesPullingTime="2.977117545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:21:47.445372639 +0000 UTC m=+541.532036402" watchObservedRunningTime="2026-09-08 16:21:47.446279042 +0000 UTC m=+541.532942789"
Sep 08 16:21:53.438122 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:21:53.438089    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-c84lt"
Sep 08 16:22:01.611298 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.611265    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l2j9c"]
Sep 08 16:22:01.614396 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.614377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l2j9c"
Sep 08 16:22:01.616633 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.616611    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-pl7bt\""
Sep 08 16:22:01.621512 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.621491    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l2j9c"]
Sep 08 16:22:01.710877 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.710843    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3cd48d13-7711-4dc1-b3b4-6940b5cb39e7-bound-sa-token\") pod \"cert-manager-55b4d954d5-l2j9c\" (UID: \"3cd48d13-7711-4dc1-b3b4-6940b5cb39e7\") " pod="cert-manager/cert-manager-55b4d954d5-l2j9c"
Sep 08 16:22:01.711053 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.710896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6clv\" (UniqueName: \"kubernetes.io/projected/3cd48d13-7711-4dc1-b3b4-6940b5cb39e7-kube-api-access-z6clv\") pod \"cert-manager-55b4d954d5-l2j9c\" (UID: \"3cd48d13-7711-4dc1-b3b4-6940b5cb39e7\") " pod="cert-manager/cert-manager-55b4d954d5-l2j9c"
Sep 08 16:22:01.811684 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.811643    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3cd48d13-7711-4dc1-b3b4-6940b5cb39e7-bound-sa-token\") pod \"cert-manager-55b4d954d5-l2j9c\" (UID: \"3cd48d13-7711-4dc1-b3b4-6940b5cb39e7\") " pod="cert-manager/cert-manager-55b4d954d5-l2j9c"
Sep 08 16:22:01.811857 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.811697    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6clv\" (UniqueName: \"kubernetes.io/projected/3cd48d13-7711-4dc1-b3b4-6940b5cb39e7-kube-api-access-z6clv\") pod \"cert-manager-55b4d954d5-l2j9c\" (UID: \"3cd48d13-7711-4dc1-b3b4-6940b5cb39e7\") " pod="cert-manager/cert-manager-55b4d954d5-l2j9c"
Sep 08 16:22:01.819471 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.819437    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3cd48d13-7711-4dc1-b3b4-6940b5cb39e7-bound-sa-token\") pod \"cert-manager-55b4d954d5-l2j9c\" (UID: \"3cd48d13-7711-4dc1-b3b4-6940b5cb39e7\") " pod="cert-manager/cert-manager-55b4d954d5-l2j9c"
Sep 08 16:22:01.819598 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.819480    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6clv\" (UniqueName: \"kubernetes.io/projected/3cd48d13-7711-4dc1-b3b4-6940b5cb39e7-kube-api-access-z6clv\") pod \"cert-manager-55b4d954d5-l2j9c\" (UID: \"3cd48d13-7711-4dc1-b3b4-6940b5cb39e7\") " pod="cert-manager/cert-manager-55b4d954d5-l2j9c"
Sep 08 16:22:01.923746 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:01.923661    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-l2j9c"
Sep 08 16:22:02.192124 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:02.192027    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-l2j9c"]
Sep 08 16:22:02.194731 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:22:02.194701    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cd48d13_7711_4dc1_b3b4_6940b5cb39e7.slice/crio-692acb0547f4336f2c259d0afcef560872e43fd8f2413d231821e206f185aae2 WatchSource:0}: Error finding container 692acb0547f4336f2c259d0afcef560872e43fd8f2413d231821e206f185aae2: Status 404 returned error can't find the container with id 692acb0547f4336f2c259d0afcef560872e43fd8f2413d231821e206f185aae2
Sep 08 16:22:02.486248 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:02.486152    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l2j9c" event={"ID":"3cd48d13-7711-4dc1-b3b4-6940b5cb39e7","Type":"ContainerStarted","Data":"763011a6cf68c979df55d20469b2c430d4f8ede54ce6f613771927f402351823"}
Sep 08 16:22:02.486248 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:02.486197    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-l2j9c" event={"ID":"3cd48d13-7711-4dc1-b3b4-6940b5cb39e7","Type":"ContainerStarted","Data":"692acb0547f4336f2c259d0afcef560872e43fd8f2413d231821e206f185aae2"}
Sep 08 16:22:02.501053 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:02.501010    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-l2j9c" podStartSLOduration=1.500995815 podStartE2EDuration="1.500995815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:02.500312024 +0000 UTC m=+556.586975774" watchObservedRunningTime="2026-09-08 16:22:02.500995815 +0000 UTC m=+556.587659562"
Sep 08 16:22:18.225755 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.225725    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"]
Sep 08 16:22:18.227915 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.227898    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.230624 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.230600    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 08 16:22:18.230624 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.230614    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 08 16:22:18.230797 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.230697    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 08 16:22:18.230960 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.230947    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 08 16:22:18.231089 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.231073    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-qjv57\""
Sep 08 16:22:18.246674 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.246647    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"]
Sep 08 16:22:18.318005 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.317982    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.318116 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.318028    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.318116 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.318046    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pzlq\" (UniqueName: \"kubernetes.io/projected/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-kube-api-access-7pzlq\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.318116 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.318066    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.318212 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.318119    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.419206 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.419167    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.419206 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.419207    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pzlq\" (UniqueName: \"kubernetes.io/projected/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-kube-api-access-7pzlq\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.419420 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.419325    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.419420 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.419374    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.419518 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.419438    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.419797 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.419771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-manifests\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.421555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.421524    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-tmp\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.421741 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.421723    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-webhook-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.421860 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.421841    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.427405 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.427386    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pzlq\" (UniqueName: \"kubernetes.io/projected/b77cf5b4-9e0c-4a24-a621-ce3396fb54cc-kube-api-access-7pzlq\") pod \"opendatahub-operator-controller-manager-68b89c556d-jqcds\" (UID: \"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc\") " pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.537927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.537870    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:18.820191 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:18.820164    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"]
Sep 08 16:22:18.822792 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:22:18.822765    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb77cf5b4_9e0c_4a24_a621_ce3396fb54cc.slice/crio-09ad34dfce2d66e8991386aafbdd5f36afd5843058c42d7eaa07f38097cba04e WatchSource:0}: Error finding container 09ad34dfce2d66e8991386aafbdd5f36afd5843058c42d7eaa07f38097cba04e: Status 404 returned error can't find the container with id 09ad34dfce2d66e8991386aafbdd5f36afd5843058c42d7eaa07f38097cba04e
Sep 08 16:22:19.543970 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:19.543926    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds" event={"ID":"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc","Type":"ContainerStarted","Data":"09ad34dfce2d66e8991386aafbdd5f36afd5843058c42d7eaa07f38097cba04e"}
Sep 08 16:22:22.557724 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:22.557689    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds" event={"ID":"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc","Type":"ContainerStarted","Data":"b3262dedf10b9db37bbbec26f0954e912f8d027c933dcc363792993039c1358b"}
Sep 08 16:22:23.562666 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:23.562627    2575 generic.go:356] "Generic (PLEG): container finished" podID="b77cf5b4-9e0c-4a24-a621-ce3396fb54cc" containerID="b3262dedf10b9db37bbbec26f0954e912f8d027c933dcc363792993039c1358b" exitCode=0
Sep 08 16:22:23.563084 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:23.562708    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds" event={"ID":"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc","Type":"ContainerDied","Data":"b3262dedf10b9db37bbbec26f0954e912f8d027c933dcc363792993039c1358b"}
Sep 08 16:22:24.568455 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:24.568419    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds" event={"ID":"b77cf5b4-9e0c-4a24-a621-ce3396fb54cc","Type":"ContainerStarted","Data":"d178edbbe07307f8f0a4cb57d9d0d08b127e22b5d35e64fb3ca4e1c3af1e54a2"}
Sep 08 16:22:24.568912 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:24.568651    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:24.591165 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:24.591120    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds" podStartSLOduration=2.138800352 podStartE2EDuration="6.591106738s" totalImagesPullingTime="3.384039858s" totalInitContainerRuntime="1.068266528s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:24.590037203 +0000 UTC m=+578.676700949" watchObservedRunningTime="2026-09-08 16:22:24.591106738 +0000 UTC m=+578.677770544"
Sep 08 16:22:35.575374 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:35.575345    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-68b89c556d-jqcds"
Sep 08 16:22:39.065529 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.065493    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7f548ddff7-sshs6"]
Sep 08 16:22:39.074874 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.074850    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.078048 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.078021    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 08 16:22:39.078160 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.078049    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 08 16:22:39.078622 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.078604    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 08 16:22:39.079188 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.079165    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 08 16:22:39.079411 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.079389    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 08 16:22:39.079676 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.079657    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dz7wm\""
Sep 08 16:22:39.085028 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.085009    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 08 16:22:39.099037 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.099008    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f548ddff7-sshs6"]
Sep 08 16:22:39.162812 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.162791    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-service-ca\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.162904 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.162831    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-oauth-config\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.162904 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.162847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-trusted-ca-bundle\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.162904 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.162882    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-config\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.163127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.162918    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlfb\" (UniqueName: \"kubernetes.io/projected/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-kube-api-access-2rlfb\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.163127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.162950    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-serving-cert\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.163127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.163011    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-oauth-serving-cert\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.263865 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.263839    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-oauth-config\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.263960 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.263868    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-trusted-ca-bundle\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.263960 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.263897    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-config\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.263960 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.263925    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlfb\" (UniqueName: \"kubernetes.io/projected/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-kube-api-access-2rlfb\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.263960 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.263958    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-serving-cert\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.264172 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.263983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-oauth-serving-cert\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.264172 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.264018    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-service-ca\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.264664 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.264643    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-config\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.264788 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.264768    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-service-ca\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.264831 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.264803    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-oauth-serving-cert\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.264969 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.264951    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-trusted-ca-bundle\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.266481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.266461    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-oauth-config\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.266600 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.266568    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-console-serving-cert\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.273022 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.272996    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlfb\" (UniqueName: \"kubernetes.io/projected/62f3cb3c-19b9-4b57-b9fe-bed995fb98ff-kube-api-access-2rlfb\") pod \"console-7f548ddff7-sshs6\" (UID: \"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff\") " pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.384972 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.384948    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:39.656878 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:39.656840    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7f548ddff7-sshs6"]
Sep 08 16:22:39.659356 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:22:39.659328    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62f3cb3c_19b9_4b57_b9fe_bed995fb98ff.slice/crio-7ebd9a9001b7db269be758777bb37e8010ce6157abd11bf1c96048b9f105c51d WatchSource:0}: Error finding container 7ebd9a9001b7db269be758777bb37e8010ce6157abd11bf1c96048b9f105c51d: Status 404 returned error can't find the container with id 7ebd9a9001b7db269be758777bb37e8010ce6157abd11bf1c96048b9f105c51d
Sep 08 16:22:40.633859 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:40.633819    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f548ddff7-sshs6" event={"ID":"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff","Type":"ContainerStarted","Data":"6c6f741ffa73529a2926a65dee8b81368aeb2f48bdc74385f68d2a4781368824"}
Sep 08 16:22:40.634239 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:40.633874    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7f548ddff7-sshs6" event={"ID":"62f3cb3c-19b9-4b57-b9fe-bed995fb98ff","Type":"ContainerStarted","Data":"7ebd9a9001b7db269be758777bb37e8010ce6157abd11bf1c96048b9f105c51d"}
Sep 08 16:22:40.651763 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:40.651708    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7f548ddff7-sshs6" podStartSLOduration=1.6516924670000002 podStartE2EDuration="1.651692467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:40.650921462 +0000 UTC m=+594.737585211" watchObservedRunningTime="2026-09-08 16:22:40.651692467 +0000 UTC m=+594.738356214"
Sep 08 16:22:43.188493 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.188456    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b99549648-ljldn"]
Sep 08 16:22:43.190963 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.190942    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.194329 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.194304    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 08 16:22:43.194605 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.194573    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-zwsrm\""
Sep 08 16:22:43.195330 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.195310    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 16:22:43.201745 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.201724    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b99549648-ljldn"]
Sep 08 16:22:43.294446 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.294421    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef99e829-0519-4df3-b0a9-17c17d399e4c-tls-certs\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.294570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.294450    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef99e829-0519-4df3-b0a9-17c17d399e4c-tmp\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.294570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.294489    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfsc\" (UniqueName: \"kubernetes.io/projected/ef99e829-0519-4df3-b0a9-17c17d399e4c-kube-api-access-9sfsc\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.395089 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.395059    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef99e829-0519-4df3-b0a9-17c17d399e4c-tls-certs\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.395197 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.395104    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef99e829-0519-4df3-b0a9-17c17d399e4c-tmp\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.395197 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.395153    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfsc\" (UniqueName: \"kubernetes.io/projected/ef99e829-0519-4df3-b0a9-17c17d399e4c-kube-api-access-9sfsc\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.397389 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.397361    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ef99e829-0519-4df3-b0a9-17c17d399e4c-tmp\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.397669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.397649    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef99e829-0519-4df3-b0a9-17c17d399e4c-tls-certs\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.404233 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.404209    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfsc\" (UniqueName: \"kubernetes.io/projected/ef99e829-0519-4df3-b0a9-17c17d399e4c-kube-api-access-9sfsc\") pod \"kube-auth-proxy-7b99549648-ljldn\" (UID: \"ef99e829-0519-4df3-b0a9-17c17d399e4c\") " pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.502100 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.502040    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn"
Sep 08 16:22:43.779340 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:43.779316    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b99549648-ljldn"]
Sep 08 16:22:43.781161 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:22:43.781138    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef99e829_0519_4df3_b0a9_17c17d399e4c.slice/crio-460736ad030e286c50445090896b97559ac96f5a226879f9518ae2d29ad67675 WatchSource:0}: Error finding container 460736ad030e286c50445090896b97559ac96f5a226879f9518ae2d29ad67675: Status 404 returned error can't find the container with id 460736ad030e286c50445090896b97559ac96f5a226879f9518ae2d29ad67675
Sep 08 16:22:44.235670 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.235637    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"]
Sep 08 16:22:44.238179 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.238162    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.240707 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.240684    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 08 16:22:44.240863 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.240734    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-pwndf\""
Sep 08 16:22:44.240863 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.240768    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 08 16:22:44.240863 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.240792    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 08 16:22:44.241031 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.240910    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 08 16:22:44.249531 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.249512    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"]
Sep 08 16:22:44.301923 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.301900    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.302013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.301928    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmb6z\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-kube-api-access-dmb6z\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.302013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.301953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.302013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.301970    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.302129 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.302056    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f149f1c0-7b2f-4321-9495-01e57c2e95f1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.302129 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.302079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.302129 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.302118    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.403360 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.403306    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.403477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.403373    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.403477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.403400    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmb6z\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-kube-api-access-dmb6z\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.403477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.403437    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.403477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.403460    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.403756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.403498    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f149f1c0-7b2f-4321-9495-01e57c2e95f1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.403756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.403520    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.404453 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.404338    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.406420 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.406395    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.406565 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.406542    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.407412 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.407373    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.407412 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.407393    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f149f1c0-7b2f-4321-9495-01e57c2e95f1-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.412091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.412071    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmb6z\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-kube-api-access-dmb6z\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.412264 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.412220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-plphg\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.547951 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.547884    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:44.652570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.652512    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn" event={"ID":"ef99e829-0519-4df3-b0a9-17c17d399e4c","Type":"ContainerStarted","Data":"460736ad030e286c50445090896b97559ac96f5a226879f9518ae2d29ad67675"}
Sep 08 16:22:44.831116 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:44.831088    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"]
Sep 08 16:22:44.833709 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:22:44.833681    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf149f1c0_7b2f_4321_9495_01e57c2e95f1.slice/crio-b34ef2371c25e0f2062985556165c63742e14f640c86f5c3c1bdb01bdca229d5 WatchSource:0}: Error finding container b34ef2371c25e0f2062985556165c63742e14f640c86f5c3c1bdb01bdca229d5: Status 404 returned error can't find the container with id b34ef2371c25e0f2062985556165c63742e14f640c86f5c3c1bdb01bdca229d5
Sep 08 16:22:45.657957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:45.657918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg" event={"ID":"f149f1c0-7b2f-4321-9495-01e57c2e95f1","Type":"ContainerStarted","Data":"b34ef2371c25e0f2062985556165c63742e14f640c86f5c3c1bdb01bdca229d5"}
Sep 08 16:22:46.427670 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:46.427644    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:22:46.428010 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:46.427994    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:22:46.434047 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:46.434027    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:22:46.434166 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:46.434104    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:22:46.450310 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:46.450287    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 08 16:22:46.666812 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:46.666770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn" event={"ID":"ef99e829-0519-4df3-b0a9-17c17d399e4c","Type":"ContainerStarted","Data":"1b0fae661d4e34873f2380327c4c9e7ec9e63b3d3547ebd5c415f0624608a4b6"}
Sep 08 16:22:46.683893 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:46.683797    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b99549648-ljldn" podStartSLOduration=1.018943212 podStartE2EDuration="3.683778708s" totalImagesPullingTime="2.664835496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:46.681653845 +0000 UTC m=+600.768317596" watchObservedRunningTime="2026-09-08 16:22:46.683778708 +0000 UTC m=+600.770442457"
Sep 08 16:22:48.180600 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:48.180554    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:22:48.180856 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:48.180651    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:22:48.679684 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:48.679647    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg" event={"ID":"f149f1c0-7b2f-4321-9495-01e57c2e95f1","Type":"ContainerStarted","Data":"b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4"}
Sep 08 16:22:48.679911 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:48.679893    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:48.707429 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:48.706218    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg" podStartSLOduration=1.361835813 podStartE2EDuration="4.706189468s" totalImagesPullingTime="3.344353655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:48.701487469 +0000 UTC m=+602.788151221" watchObservedRunningTime="2026-09-08 16:22:48.706189468 +0000 UTC m=+602.792853219"
Sep 08 16:22:49.385703 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:49.385672    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:49.386946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:49.385768    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:49.390560 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:49.390533    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:49.686335 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:49.686255    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:49.688169 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:49.688149    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7f548ddff7-sshs6"
Sep 08 16:22:56.685550 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.685515    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"]
Sep 08 16:22:56.687775 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.687759    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.710430 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.710400    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"]
Sep 08 16:22:56.808650 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.808622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.808777 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.808663    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.808777 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.808722    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.808777 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.808755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.808893 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.808790    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.808893 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.808821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zgf\" (UniqueName: \"kubernetes.io/projected/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-kube-api-access-b9zgf\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.808893 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.808847    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.909668 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.909627    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.909944 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.909925    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.910402 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.910380    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.910875 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.910853    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.911711 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.911690    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.912138 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.911633    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.912400 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.912354    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zgf\" (UniqueName: \"kubernetes.io/projected/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-kube-api-access-b9zgf\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.912556 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.912521    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.913217 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.913193    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.914185 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.914159    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.914746 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.914718    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.915793 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.915772    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.918118 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.918093    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.921896 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.921876    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zgf\" (UniqueName: \"kubernetes.io/projected/4c88f4ae-d8bc-4be8-9228-d27b5ce35837-kube-api-access-b9zgf\") pod \"istiod-openshift-gateway-5677ccb49b-pgh8w\" (UID: \"4c88f4ae-d8bc-4be8-9228-d27b5ce35837\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:56.996664 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:56.996574    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:57.273334 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:57.273312    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"]
Sep 08 16:22:57.275904 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:22:57.275866    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c88f4ae_d8bc_4be8_9228_d27b5ce35837.slice/crio-d6c1e6c96d8d9bfea550dcc7f1bb379e9b3f0a277051dbed17cd1c806bb1eac4 WatchSource:0}: Error finding container d6c1e6c96d8d9bfea550dcc7f1bb379e9b3f0a277051dbed17cd1c806bb1eac4: Status 404 returned error can't find the container with id d6c1e6c96d8d9bfea550dcc7f1bb379e9b3f0a277051dbed17cd1c806bb1eac4
Sep 08 16:22:57.278985 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:57.278953    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:22:57.279075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:57.279026    2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 08 16:22:57.727883 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:57.727843    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w" event={"ID":"4c88f4ae-d8bc-4be8-9228-d27b5ce35837","Type":"ContainerStarted","Data":"c971d46849505339525d43fdf1c72e4b713ec07da42ef9d870225b4b0ac27d96"}
Sep 08 16:22:57.728350 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:57.727937    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w" event={"ID":"4c88f4ae-d8bc-4be8-9228-d27b5ce35837","Type":"ContainerStarted","Data":"d6c1e6c96d8d9bfea550dcc7f1bb379e9b3f0a277051dbed17cd1c806bb1eac4"}
Sep 08 16:22:57.728703 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:57.728678    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:57.753910 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:57.753862    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w" podStartSLOduration=1.7538480189999999 podStartE2EDuration="1.753848019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:22:57.752248852 +0000 UTC m=+611.838912601" watchObservedRunningTime="2026-09-08 16:22:57.753848019 +0000 UTC m=+611.840511766"
Sep 08 16:22:58.733559 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:58.733526    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-pgh8w"
Sep 08 16:22:58.871563 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:58.871523    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"]
Sep 08 16:22:58.872667 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:58.872608    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg" podUID="f149f1c0-7b2f-4321-9495-01e57c2e95f1" containerName="discovery" containerID="cri-o://b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4" gracePeriod=30
Sep 08 16:22:59.127027 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.126996    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:59.235303 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.235264    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmb6z\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-kube-api-access-dmb6z\") pod \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") "
Sep 08 16:22:59.235481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.235319    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-cacerts\") pod \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") "
Sep 08 16:22:59.235481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.235366    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-token\") pod \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") "
Sep 08 16:22:59.235481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.235396    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-dns-cert\") pod \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") "
Sep 08 16:22:59.235481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.235446    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-ca-configmap\") pod \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") "
Sep 08 16:22:59.235481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.235469    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f149f1c0-7b2f-4321-9495-01e57c2e95f1-local-certs\") pod \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") "
Sep 08 16:22:59.235772 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.235492    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-kubeconfig\") pod \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\" (UID: \"f149f1c0-7b2f-4321-9495-01e57c2e95f1\") "
Sep 08 16:22:59.236142 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.236113    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f149f1c0-7b2f-4321-9495-01e57c2e95f1" (UID: "f149f1c0-7b2f-4321-9495-01e57c2e95f1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:22:59.238039 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.237998    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f149f1c0-7b2f-4321-9495-01e57c2e95f1" (UID: "f149f1c0-7b2f-4321-9495-01e57c2e95f1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:22:59.238379 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.238181    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f149f1c0-7b2f-4321-9495-01e57c2e95f1-local-certs" (OuterVolumeSpecName: "local-certs") pod "f149f1c0-7b2f-4321-9495-01e57c2e95f1" (UID: "f149f1c0-7b2f-4321-9495-01e57c2e95f1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:22:59.238818 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.238798    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "f149f1c0-7b2f-4321-9495-01e57c2e95f1" (UID: "f149f1c0-7b2f-4321-9495-01e57c2e95f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:22:59.239282 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.239253    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f149f1c0-7b2f-4321-9495-01e57c2e95f1" (UID: "f149f1c0-7b2f-4321-9495-01e57c2e95f1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:22:59.239381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.239345    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-cacerts" (OuterVolumeSpecName: "cacerts") pod "f149f1c0-7b2f-4321-9495-01e57c2e95f1" (UID: "f149f1c0-7b2f-4321-9495-01e57c2e95f1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:22:59.239381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.239365    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-kube-api-access-dmb6z" (OuterVolumeSpecName: "kube-api-access-dmb6z") pod "f149f1c0-7b2f-4321-9495-01e57c2e95f1" (UID: "f149f1c0-7b2f-4321-9495-01e57c2e95f1"). InnerVolumeSpecName "kube-api-access-dmb6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:22:59.240756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.240705    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"]
Sep 08 16:22:59.241007 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.240994    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f149f1c0-7b2f-4321-9495-01e57c2e95f1" containerName="discovery"
Sep 08 16:22:59.241007 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.241007    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f149f1c0-7b2f-4321-9495-01e57c2e95f1" containerName="discovery"
Sep 08 16:22:59.241154 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.241053    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f149f1c0-7b2f-4321-9495-01e57c2e95f1" containerName="discovery"
Sep 08 16:22:59.243087 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.243067    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:22:59.246435 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.246396    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 08 16:22:59.246763 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.246747    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r56g5\""
Sep 08 16:22:59.258233 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.258213    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"]
Sep 08 16:22:59.336862 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336839    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntwcc\" (UniqueName: \"kubernetes.io/projected/44279791-073f-4837-b065-4550a8ba5fca-kube-api-access-ntwcc\") pod \"llmisvc-controller-manager-777c6bd7d5-h9nxn\" (UID: \"44279791-073f-4837-b065-4550a8ba5fca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:22:59.336995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336872    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44279791-073f-4837-b065-4550a8ba5fca-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h9nxn\" (UID: \"44279791-073f-4837-b065-4550a8ba5fca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:22:59.336995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336918    2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-ca-configmap\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:22:59.336995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336929    2575 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f149f1c0-7b2f-4321-9495-01e57c2e95f1-local-certs\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:22:59.336995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336938    2575 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-kubeconfig\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:22:59.336995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336945    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmb6z\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-kube-api-access-dmb6z\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:22:59.336995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336953    2575 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-cacerts\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:22:59.336995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336961    2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-token\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:22:59.336995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.336969    2575 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f149f1c0-7b2f-4321-9495-01e57c2e95f1-istio-csr-dns-cert\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:22:59.437948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.437922    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntwcc\" (UniqueName: \"kubernetes.io/projected/44279791-073f-4837-b065-4550a8ba5fca-kube-api-access-ntwcc\") pod \"llmisvc-controller-manager-777c6bd7d5-h9nxn\" (UID: \"44279791-073f-4837-b065-4550a8ba5fca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:22:59.438082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.437953    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44279791-073f-4837-b065-4550a8ba5fca-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h9nxn\" (UID: \"44279791-073f-4837-b065-4550a8ba5fca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:22:59.440238 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.440220    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/44279791-073f-4837-b065-4550a8ba5fca-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-h9nxn\" (UID: \"44279791-073f-4837-b065-4550a8ba5fca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:22:59.456153 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.456130    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntwcc\" (UniqueName: \"kubernetes.io/projected/44279791-073f-4837-b065-4550a8ba5fca-kube-api-access-ntwcc\") pod \"llmisvc-controller-manager-777c6bd7d5-h9nxn\" (UID: \"44279791-073f-4837-b065-4550a8ba5fca\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:22:59.562714 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.562662    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:22:59.737405 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.737375    2575 generic.go:356] "Generic (PLEG): container finished" podID="f149f1c0-7b2f-4321-9495-01e57c2e95f1" containerID="b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4" exitCode=0
Sep 08 16:22:59.737830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.737412    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"
Sep 08 16:22:59.737830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.737457    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg" event={"ID":"f149f1c0-7b2f-4321-9495-01e57c2e95f1","Type":"ContainerDied","Data":"b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4"}
Sep 08 16:22:59.737830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.737505    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg" event={"ID":"f149f1c0-7b2f-4321-9495-01e57c2e95f1","Type":"ContainerDied","Data":"b34ef2371c25e0f2062985556165c63742e14f640c86f5c3c1bdb01bdca229d5"}
Sep 08 16:22:59.737830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.737531    2575 scope.go:117] "RemoveContainer" containerID="b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4"
Sep 08 16:22:59.745971 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.745954    2575 scope.go:117] "RemoveContainer" containerID="b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4"
Sep 08 16:22:59.746212 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:22:59.746195    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4\": container with ID starting with b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4 not found: ID does not exist" containerID="b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4"
Sep 08 16:22:59.746280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.746218    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4"} err="failed to get container status \"b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4\": rpc error: code = NotFound desc = could not find container \"b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4\": container with ID starting with b621c07a819053c8ad99c52774175895b1ba56af4cc577379f0f86dd921588a4 not found: ID does not exist"
Sep 08 16:22:59.759496 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.759472    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"]
Sep 08 16:22:59.763302 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.763281    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-plphg"]
Sep 08 16:22:59.831561 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:22:59.831539    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"]
Sep 08 16:22:59.834598 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:22:59.834564    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod44279791_073f_4837_b065_4550a8ba5fca.slice/crio-1f3f253f7f11b4e593856f85271c93da6ffa3950327712c7dd779927c43126c3 WatchSource:0}: Error finding container 1f3f253f7f11b4e593856f85271c93da6ffa3950327712c7dd779927c43126c3: Status 404 returned error can't find the container with id 1f3f253f7f11b4e593856f85271c93da6ffa3950327712c7dd779927c43126c3
Sep 08 16:23:00.506687 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:23:00.506658    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f149f1c0-7b2f-4321-9495-01e57c2e95f1" path="/var/lib/kubelet/pods/f149f1c0-7b2f-4321-9495-01e57c2e95f1/volumes"
Sep 08 16:23:00.742636 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:23:00.742607    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn" event={"ID":"44279791-073f-4837-b065-4550a8ba5fca","Type":"ContainerStarted","Data":"1f3f253f7f11b4e593856f85271c93da6ffa3950327712c7dd779927c43126c3"}
Sep 08 16:23:02.753992 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:23:02.753956    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn" event={"ID":"44279791-073f-4837-b065-4550a8ba5fca","Type":"ContainerStarted","Data":"a32f7dc16c6b14538fdefd3319079bee3bfb66573def89a86a65852ec82dc011"}
Sep 08 16:23:02.754353 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:23:02.754103    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:23:02.773611 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:23:02.773545    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn" podStartSLOduration=1.018324122 podStartE2EDuration="3.77352926s" totalImagesPullingTime="2.755205138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:23:02.772437398 +0000 UTC m=+616.859101145" watchObservedRunningTime="2026-09-08 16:23:02.77352926 +0000 UTC m=+616.860193007"
Sep 08 16:23:33.760344 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:23:33.760316    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-h9nxn"
Sep 08 16:24:11.558505 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.558474    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kj49v"]
Sep 08 16:24:11.560354 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.560338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-kj49v"
Sep 08 16:24:11.562911 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.562883    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 08 16:24:11.563027 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.563004    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rmdcv\""
Sep 08 16:24:11.563087 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.563042    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 08 16:24:11.573060 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.573039    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kj49v"]
Sep 08 16:24:11.688090 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.688059    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69fhc\" (UniqueName: \"kubernetes.io/projected/fcc869a4-93d8-4ea3-ba93-ab7de1cd8797-kube-api-access-69fhc\") pod \"authorino-operator-76b478bd78-kj49v\" (UID: \"fcc869a4-93d8-4ea3-ba93-ab7de1cd8797\") " pod="kuadrant-system/authorino-operator-76b478bd78-kj49v"
Sep 08 16:24:11.789345 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.789310    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69fhc\" (UniqueName: \"kubernetes.io/projected/fcc869a4-93d8-4ea3-ba93-ab7de1cd8797-kube-api-access-69fhc\") pod \"authorino-operator-76b478bd78-kj49v\" (UID: \"fcc869a4-93d8-4ea3-ba93-ab7de1cd8797\") " pod="kuadrant-system/authorino-operator-76b478bd78-kj49v"
Sep 08 16:24:11.802075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.802055    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69fhc\" (UniqueName: \"kubernetes.io/projected/fcc869a4-93d8-4ea3-ba93-ab7de1cd8797-kube-api-access-69fhc\") pod \"authorino-operator-76b478bd78-kj49v\" (UID: \"fcc869a4-93d8-4ea3-ba93-ab7de1cd8797\") " pod="kuadrant-system/authorino-operator-76b478bd78-kj49v"
Sep 08 16:24:11.870633 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:11.870608    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-kj49v"
Sep 08 16:24:12.140280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:12.140257    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-kj49v"]
Sep 08 16:24:12.143042 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:24:12.143017    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc869a4_93d8_4ea3_ba93_ab7de1cd8797.slice/crio-acd006adeabca45fdf5d93081f7dc0df063f4715c3e5713f69d0bb0deb14d34d WatchSource:0}: Error finding container acd006adeabca45fdf5d93081f7dc0df063f4715c3e5713f69d0bb0deb14d34d: Status 404 returned error can't find the container with id acd006adeabca45fdf5d93081f7dc0df063f4715c3e5713f69d0bb0deb14d34d
Sep 08 16:24:13.055162 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:13.055125    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-kj49v" event={"ID":"fcc869a4-93d8-4ea3-ba93-ab7de1cd8797","Type":"ContainerStarted","Data":"acd006adeabca45fdf5d93081f7dc0df063f4715c3e5713f69d0bb0deb14d34d"}
Sep 08 16:24:14.287178 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.287141    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"]
Sep 08 16:24:14.291931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.291896    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"
Sep 08 16:24:14.294530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.294502    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 08 16:24:14.294715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.294502    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-pfpks\""
Sep 08 16:24:14.303769 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.303717    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"]
Sep 08 16:24:14.412305 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.412258    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j7dx\" (UniqueName: \"kubernetes.io/projected/dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f-kube-api-access-9j7dx\") pod \"dns-operator-controller-manager-7b84cd7c9c-w59v2\" (UID: \"dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"
Sep 08 16:24:14.513229 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.513191    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j7dx\" (UniqueName: \"kubernetes.io/projected/dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f-kube-api-access-9j7dx\") pod \"dns-operator-controller-manager-7b84cd7c9c-w59v2\" (UID: \"dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"
Sep 08 16:24:14.521706 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.521669    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j7dx\" (UniqueName: \"kubernetes.io/projected/dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f-kube-api-access-9j7dx\") pod \"dns-operator-controller-manager-7b84cd7c9c-w59v2\" (UID: \"dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"
Sep 08 16:24:14.607128 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:14.607085    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"
Sep 08 16:24:15.406141 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:15.406117    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"]
Sep 08 16:24:15.408504 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:24:15.408475    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb0e707_dbd4_4aad_b7bc_6afb4ac8de7f.slice/crio-d283db4c42a4e1a2327adf736bebd1f286274a5dc1e8c5ee1fe716dc043c8be9 WatchSource:0}: Error finding container d283db4c42a4e1a2327adf736bebd1f286274a5dc1e8c5ee1fe716dc043c8be9: Status 404 returned error can't find the container with id d283db4c42a4e1a2327adf736bebd1f286274a5dc1e8c5ee1fe716dc043c8be9
Sep 08 16:24:16.071499 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:16.071458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-kj49v" event={"ID":"fcc869a4-93d8-4ea3-ba93-ab7de1cd8797","Type":"ContainerStarted","Data":"3dcfd52d4b173451b2bcda97e4cef5815202071186065028f07bd2035cff3506"}
Sep 08 16:24:16.071719 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:16.071571    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-kj49v"
Sep 08 16:24:16.072824 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:16.072803    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2" event={"ID":"dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f","Type":"ContainerStarted","Data":"d283db4c42a4e1a2327adf736bebd1f286274a5dc1e8c5ee1fe716dc043c8be9"}
Sep 08 16:24:16.101328 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:16.101291    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-kj49v" podStartSLOduration=2.082763954 podStartE2EDuration="5.101277934s" totalImagesPullingTime="3.01851398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:16.099466423 +0000 UTC m=+690.186130170" watchObservedRunningTime="2026-09-08 16:24:16.101277934 +0000 UTC m=+690.187941681"
Sep 08 16:24:18.086752 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.086715    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2" event={"ID":"dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f","Type":"ContainerStarted","Data":"10c83916139507c7f526039d759820d39690c153caf997b19c4e3e14350d0771"}
Sep 08 16:24:18.087217 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.086830    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"
Sep 08 16:24:18.119180 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.119125    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2" podStartSLOduration=2.397435421 podStartE2EDuration="4.119109587s" totalImagesPullingTime="1.721674166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:18.117922747 +0000 UTC m=+692.204586497" watchObservedRunningTime="2026-09-08 16:24:18.119109587 +0000 UTC m=+692.205773334"
Sep 08 16:24:18.714755 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.714724    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"]
Sep 08 16:24:18.717264 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.717247    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:18.719462 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.719441    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sm84v\""
Sep 08 16:24:18.729039 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.729020    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"]
Sep 08 16:24:18.852297 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.852257    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn4gw\" (UniqueName: \"kubernetes.io/projected/20eaad0c-727a-4509-b528-d7e61452508b-kube-api-access-cn4gw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" (UID: \"20eaad0c-727a-4509-b528-d7e61452508b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:18.852466 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.852343    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20eaad0c-727a-4509-b528-d7e61452508b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" (UID: \"20eaad0c-727a-4509-b528-d7e61452508b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:18.953133 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.953102    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn4gw\" (UniqueName: \"kubernetes.io/projected/20eaad0c-727a-4509-b528-d7e61452508b-kube-api-access-cn4gw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" (UID: \"20eaad0c-727a-4509-b528-d7e61452508b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:18.953266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.953161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20eaad0c-727a-4509-b528-d7e61452508b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" (UID: \"20eaad0c-727a-4509-b528-d7e61452508b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:18.953477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.953461    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20eaad0c-727a-4509-b528-d7e61452508b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" (UID: \"20eaad0c-727a-4509-b528-d7e61452508b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:18.969981 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:18.969914    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn4gw\" (UniqueName: \"kubernetes.io/projected/20eaad0c-727a-4509-b528-d7e61452508b-kube-api-access-cn4gw\") pod \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" (UID: \"20eaad0c-727a-4509-b528-d7e61452508b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:19.026920 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:19.026891    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:19.307501 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:19.307472    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"]
Sep 08 16:24:19.309763 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:24:19.309739    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20eaad0c_727a_4509_b528_d7e61452508b.slice/crio-74a74386292c5de4e34bbc34b24e4f02d1292e3eae7933f6beb37d4da3544fdb WatchSource:0}: Error finding container 74a74386292c5de4e34bbc34b24e4f02d1292e3eae7933f6beb37d4da3544fdb: Status 404 returned error can't find the container with id 74a74386292c5de4e34bbc34b24e4f02d1292e3eae7933f6beb37d4da3544fdb
Sep 08 16:24:20.098931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:20.098899    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t" event={"ID":"20eaad0c-727a-4509-b528-d7e61452508b","Type":"ContainerStarted","Data":"74a74386292c5de4e34bbc34b24e4f02d1292e3eae7933f6beb37d4da3544fdb"}
Sep 08 16:24:23.117891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:23.117854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t" event={"ID":"20eaad0c-727a-4509-b528-d7e61452508b","Type":"ContainerStarted","Data":"492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f"}
Sep 08 16:24:23.118356 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:23.117984    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:23.143511 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:23.143465    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t" podStartSLOduration=1.674971444 podStartE2EDuration="5.143449784s" totalImagesPullingTime="3.46847834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:23.141261401 +0000 UTC m=+697.227925146" watchObservedRunningTime="2026-09-08 16:24:23.143449784 +0000 UTC m=+697.230113531"
Sep 08 16:24:27.082635 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:27.082599    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-kj49v"
Sep 08 16:24:29.095392 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:29.095364    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-w59v2"
Sep 08 16:24:34.123809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:34.123778    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:36.192842 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.192800    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"]
Sep 08 16:24:36.193451 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.193051    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t" podUID="20eaad0c-727a-4509-b528-d7e61452508b" containerName="manager" containerID="cri-o://492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f" gracePeriod=2
Sep 08 16:24:36.201884 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.201566    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"]
Sep 08 16:24:36.203780 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:24:36.203741    2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"20eaad0c-727a-4509-b528-d7e61452508b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-9md5t\": pods \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:36.243631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.243606    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"]
Sep 08 16:24:36.244113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.244086    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20eaad0c-727a-4509-b528-d7e61452508b" containerName="manager"
Sep 08 16:24:36.244113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.244112    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="20eaad0c-727a-4509-b528-d7e61452508b" containerName="manager"
Sep 08 16:24:36.244244 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.244172    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="20eaad0c-727a-4509-b528-d7e61452508b" containerName="manager"
Sep 08 16:24:36.246083 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.246065    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:36.248364 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:24:36.248334    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" is forbidden: User \"system:node:ip-10-0-128-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-92.ec2.internal' and this object" podUID="20eaad0c-727a-4509-b528-d7e61452508b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:36.281328 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.281298    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"]
Sep 08 16:24:36.398242 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.398204    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z9gl\" (UniqueName: \"kubernetes.io/projected/94ac343c-62c3-4216-be00-f58a557170ab-kube-api-access-6z9gl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gx4v4\" (UID: \"94ac343c-62c3-4216-be00-f58a557170ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:36.398364 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.398259    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ac343c-62c3-4216-be00-f58a557170ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gx4v4\" (UID: \"94ac343c-62c3-4216-be00-f58a557170ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:36.427229 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.427208    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:36.429853 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:24:36.429822    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" is forbidden: User \"system:node:ip-10-0-128-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-92.ec2.internal' and this object" podUID="20eaad0c-727a-4509-b528-d7e61452508b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:36.498925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.498866    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z9gl\" (UniqueName: \"kubernetes.io/projected/94ac343c-62c3-4216-be00-f58a557170ab-kube-api-access-6z9gl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gx4v4\" (UID: \"94ac343c-62c3-4216-be00-f58a557170ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:36.498925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.498895    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ac343c-62c3-4216-be00-f58a557170ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gx4v4\" (UID: \"94ac343c-62c3-4216-be00-f58a557170ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:36.499182 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.499167    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ac343c-62c3-4216-be00-f58a557170ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gx4v4\" (UID: \"94ac343c-62c3-4216-be00-f58a557170ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:36.504240 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:24:36.504214    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-9md5t\" is forbidden: User \"system:node:ip-10-0-128-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-92.ec2.internal' and this object" podUID="20eaad0c-727a-4509-b528-d7e61452508b" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:36.510222 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.510199    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z9gl\" (UniqueName: \"kubernetes.io/projected/94ac343c-62c3-4216-be00-f58a557170ab-kube-api-access-6z9gl\") pod \"kuadrant-operator-controller-manager-76f8f686bf-gx4v4\" (UID: \"94ac343c-62c3-4216-be00-f58a557170ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:36.585158 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.585134    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:36.599858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.599835    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20eaad0c-727a-4509-b528-d7e61452508b-extensions-socket-volume\") pod \"20eaad0c-727a-4509-b528-d7e61452508b\" (UID: \"20eaad0c-727a-4509-b528-d7e61452508b\") "
Sep 08 16:24:36.599958 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.599896    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn4gw\" (UniqueName: \"kubernetes.io/projected/20eaad0c-727a-4509-b528-d7e61452508b-kube-api-access-cn4gw\") pod \"20eaad0c-727a-4509-b528-d7e61452508b\" (UID: \"20eaad0c-727a-4509-b528-d7e61452508b\") "
Sep 08 16:24:36.600286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.600263    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20eaad0c-727a-4509-b528-d7e61452508b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "20eaad0c-727a-4509-b528-d7e61452508b" (UID: "20eaad0c-727a-4509-b528-d7e61452508b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:24:36.601885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.601863    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20eaad0c-727a-4509-b528-d7e61452508b-kube-api-access-cn4gw" (OuterVolumeSpecName: "kube-api-access-cn4gw") pod "20eaad0c-727a-4509-b528-d7e61452508b" (UID: "20eaad0c-727a-4509-b528-d7e61452508b"). InnerVolumeSpecName "kube-api-access-cn4gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:24:36.701142 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.701114    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/20eaad0c-727a-4509-b528-d7e61452508b-extensions-socket-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:24:36.701142 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.701141    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn4gw\" (UniqueName: \"kubernetes.io/projected/20eaad0c-727a-4509-b528-d7e61452508b-kube-api-access-cn4gw\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:24:36.865138 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:36.865108    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"]
Sep 08 16:24:36.868647 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:24:36.868621    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ac343c_62c3_4216_be00_f58a557170ab.slice/crio-da0f0642a3ba1e6ab3725e8d2ded32e274b3e67cd044924b9b4fd1af6abe72ca WatchSource:0}: Error finding container da0f0642a3ba1e6ab3725e8d2ded32e274b3e67cd044924b9b4fd1af6abe72ca: Status 404 returned error can't find the container with id da0f0642a3ba1e6ab3725e8d2ded32e274b3e67cd044924b9b4fd1af6abe72ca
Sep 08 16:24:37.182701 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.182625    2575 generic.go:356] "Generic (PLEG): container finished" podID="20eaad0c-727a-4509-b528-d7e61452508b" containerID="492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f" exitCode=0
Sep 08 16:24:37.182701 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.182666    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-9md5t"
Sep 08 16:24:37.182891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.182726    2575 scope.go:117] "RemoveContainer" containerID="492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f"
Sep 08 16:24:37.184726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.184705    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4" event={"ID":"94ac343c-62c3-4216-be00-f58a557170ab","Type":"ContainerStarted","Data":"f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765"}
Sep 08 16:24:37.184847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.184755    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4" event={"ID":"94ac343c-62c3-4216-be00-f58a557170ab","Type":"ContainerStarted","Data":"da0f0642a3ba1e6ab3725e8d2ded32e274b3e67cd044924b9b4fd1af6abe72ca"}
Sep 08 16:24:37.184847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.184801    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:37.191184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.191167    2575 scope.go:117] "RemoveContainer" containerID="492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f"
Sep 08 16:24:37.191438 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:24:37.191413    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f\": container with ID starting with 492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f not found: ID does not exist" containerID="492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f"
Sep 08 16:24:37.191495 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.191445    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f"} err="failed to get container status \"492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f\": rpc error: code = NotFound desc = could not find container \"492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f\": container with ID starting with 492a65db659fc9c3139342e882a26e36f8142b365ba08f0dbfd9805db800bf8f not found: ID does not exist"
Sep 08 16:24:37.214004 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:37.213957    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4" podStartSLOduration=1.213942407 podStartE2EDuration="1.213942407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:37.212062974 +0000 UTC m=+711.298726721" watchObservedRunningTime="2026-09-08 16:24:37.213942407 +0000 UTC m=+711.300606159"
Sep 08 16:24:38.506362 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:38.506324    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20eaad0c-727a-4509-b528-d7e61452508b" path="/var/lib/kubelet/pods/20eaad0c-727a-4509-b528-d7e61452508b/volumes"
Sep 08 16:24:48.193483 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:48.193452    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:51.681931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:51.681894    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"]
Sep 08 16:24:51.682364 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:51.682111    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4" podUID="94ac343c-62c3-4216-be00-f58a557170ab" containerName="manager" containerID="cri-o://f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765" gracePeriod=10
Sep 08 16:24:51.929368 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:51.929345    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"]
Sep 08 16:24:51.949264 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:51.949213    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"]
Sep 08 16:24:51.949346 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:51.949295    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:51.959307 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:51.959283    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:52.007970 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.007946    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6z9gl\" (UniqueName: \"kubernetes.io/projected/94ac343c-62c3-4216-be00-f58a557170ab-kube-api-access-6z9gl\") pod \"94ac343c-62c3-4216-be00-f58a557170ab\" (UID: \"94ac343c-62c3-4216-be00-f58a557170ab\") "
Sep 08 16:24:52.008075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.007985    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ac343c-62c3-4216-be00-f58a557170ab-extensions-socket-volume\") pod \"94ac343c-62c3-4216-be00-f58a557170ab\" (UID: \"94ac343c-62c3-4216-be00-f58a557170ab\") "
Sep 08 16:24:52.008159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.008143    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kfqxv\" (UID: \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:52.008315 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.008267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dksfx\" (UniqueName: \"kubernetes.io/projected/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-kube-api-access-dksfx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kfqxv\" (UID: \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:52.008429 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.008409    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ac343c-62c3-4216-be00-f58a557170ab-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "94ac343c-62c3-4216-be00-f58a557170ab" (UID: "94ac343c-62c3-4216-be00-f58a557170ab"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:24:52.010843 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.010815    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ac343c-62c3-4216-be00-f58a557170ab-kube-api-access-6z9gl" (OuterVolumeSpecName: "kube-api-access-6z9gl") pod "94ac343c-62c3-4216-be00-f58a557170ab" (UID: "94ac343c-62c3-4216-be00-f58a557170ab"). InnerVolumeSpecName "kube-api-access-6z9gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:24:52.109316 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.109293    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dksfx\" (UniqueName: \"kubernetes.io/projected/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-kube-api-access-dksfx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kfqxv\" (UID: \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:52.109425 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.109356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kfqxv\" (UID: \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:52.109425 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.109398    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z9gl\" (UniqueName: \"kubernetes.io/projected/94ac343c-62c3-4216-be00-f58a557170ab-kube-api-access-6z9gl\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:24:52.109425 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.109408    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/94ac343c-62c3-4216-be00-f58a557170ab-extensions-socket-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:24:52.109684 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.109669    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kfqxv\" (UID: \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:52.117954 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.117932    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dksfx\" (UniqueName: \"kubernetes.io/projected/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-kube-api-access-dksfx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-kfqxv\" (UID: \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:52.258191 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.258134    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:52.259253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.259151    2575 generic.go:356] "Generic (PLEG): container finished" podID="94ac343c-62c3-4216-be00-f58a557170ab" containerID="f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765" exitCode=0
Sep 08 16:24:52.259253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.259205    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"
Sep 08 16:24:52.259253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.259234    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4" event={"ID":"94ac343c-62c3-4216-be00-f58a557170ab","Type":"ContainerDied","Data":"f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765"}
Sep 08 16:24:52.259405 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.259304    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4" event={"ID":"94ac343c-62c3-4216-be00-f58a557170ab","Type":"ContainerDied","Data":"da0f0642a3ba1e6ab3725e8d2ded32e274b3e67cd044924b9b4fd1af6abe72ca"}
Sep 08 16:24:52.259405 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.259326    2575 scope.go:117] "RemoveContainer" containerID="f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765"
Sep 08 16:24:52.268119 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.268086    2575 scope.go:117] "RemoveContainer" containerID="f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765"
Sep 08 16:24:52.268429 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:24:52.268397    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765\": container with ID starting with f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765 not found: ID does not exist" containerID="f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765"
Sep 08 16:24:52.268481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.268442    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765"} err="failed to get container status \"f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765\": rpc error: code = NotFound desc = could not find container \"f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765\": container with ID starting with f84f100142324b9108cde9b93e1e985c931b20045c4a88a674821cd6e2fd0765 not found: ID does not exist"
Sep 08 16:24:52.284512 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.284491    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"]
Sep 08 16:24:52.290311 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.290291    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-gx4v4"]
Sep 08 16:24:52.507652 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.507622    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ac343c-62c3-4216-be00-f58a557170ab" path="/var/lib/kubelet/pods/94ac343c-62c3-4216-be00-f58a557170ab/volumes"
Sep 08 16:24:52.536485 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:52.536427    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"]
Sep 08 16:24:52.540566 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:24:52.540543    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45da06f9_39aa_44fd_bbcf_5e9fb10a298f.slice/crio-042c280f488755078f6a7cd28ca7f848c1ce2b5ac49e03f725c3d9ac2dca2e00 WatchSource:0}: Error finding container 042c280f488755078f6a7cd28ca7f848c1ce2b5ac49e03f725c3d9ac2dca2e00: Status 404 returned error can't find the container with id 042c280f488755078f6a7cd28ca7f848c1ce2b5ac49e03f725c3d9ac2dca2e00
Sep 08 16:24:53.267516 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:53.267480    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv" event={"ID":"45da06f9-39aa-44fd-bbcf-5e9fb10a298f","Type":"ContainerStarted","Data":"17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721"}
Sep 08 16:24:53.267877 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:53.267538    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv" event={"ID":"45da06f9-39aa-44fd-bbcf-5e9fb10a298f","Type":"ContainerStarted","Data":"042c280f488755078f6a7cd28ca7f848c1ce2b5ac49e03f725c3d9ac2dca2e00"}
Sep 08 16:24:53.267877 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:53.267615    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:24:53.294646 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:24:53.294600    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv" podStartSLOduration=2.294572696 podStartE2EDuration="2.294572696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:24:53.293833094 +0000 UTC m=+727.380496845" watchObservedRunningTime="2026-09-08 16:24:53.294572696 +0000 UTC m=+727.381236443"
Sep 08 16:25:04.275221 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:04.275189    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:25:21.374837 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.374802    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7hszn"]
Sep 08 16:25:21.375279 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.375119    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ac343c-62c3-4216-be00-f58a557170ab" containerName="manager"
Sep 08 16:25:21.375279 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.375129    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ac343c-62c3-4216-be00-f58a557170ab" containerName="manager"
Sep 08 16:25:21.375279 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.375199    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ac343c-62c3-4216-be00-f58a557170ab" containerName="manager"
Sep 08 16:25:21.377069 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.377048    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:21.380657 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.380637    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 08 16:25:21.380773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.380640    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-5p6kn\""
Sep 08 16:25:21.389370 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.389347    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7hszn"]
Sep 08 16:25:21.447648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.447614    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f066f35-4ec8-4873-8fe5-e437340b85c1-config-file\") pod \"limitador-limitador-8588749977-7hszn\" (UID: \"8f066f35-4ec8-4873-8fe5-e437340b85c1\") " pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:21.447813 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.447735    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b42d8\" (UniqueName: \"kubernetes.io/projected/8f066f35-4ec8-4873-8fe5-e437340b85c1-kube-api-access-b42d8\") pod \"limitador-limitador-8588749977-7hszn\" (UID: \"8f066f35-4ec8-4873-8fe5-e437340b85c1\") " pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:21.475664 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.475634    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7hszn"]
Sep 08 16:25:21.548404 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.548371    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b42d8\" (UniqueName: \"kubernetes.io/projected/8f066f35-4ec8-4873-8fe5-e437340b85c1-kube-api-access-b42d8\") pod \"limitador-limitador-8588749977-7hszn\" (UID: \"8f066f35-4ec8-4873-8fe5-e437340b85c1\") " pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:21.548550 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.548450    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f066f35-4ec8-4873-8fe5-e437340b85c1-config-file\") pod \"limitador-limitador-8588749977-7hszn\" (UID: \"8f066f35-4ec8-4873-8fe5-e437340b85c1\") " pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:21.549062 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.549043    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f066f35-4ec8-4873-8fe5-e437340b85c1-config-file\") pod \"limitador-limitador-8588749977-7hszn\" (UID: \"8f066f35-4ec8-4873-8fe5-e437340b85c1\") " pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:21.557073 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.557047    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b42d8\" (UniqueName: \"kubernetes.io/projected/8f066f35-4ec8-4873-8fe5-e437340b85c1-kube-api-access-b42d8\") pod \"limitador-limitador-8588749977-7hszn\" (UID: \"8f066f35-4ec8-4873-8fe5-e437340b85c1\") " pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:21.687370 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.687288    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:21.956749 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:21.956594    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7hszn"]
Sep 08 16:25:21.959695 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:25:21.959668    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f066f35_4ec8_4873_8fe5_e437340b85c1.slice/crio-904cb7dfbdc0ab0f3aa838a89d70cd1a5aedabcb1177ac208503522232f80c26 WatchSource:0}: Error finding container 904cb7dfbdc0ab0f3aa838a89d70cd1a5aedabcb1177ac208503522232f80c26: Status 404 returned error can't find the container with id 904cb7dfbdc0ab0f3aa838a89d70cd1a5aedabcb1177ac208503522232f80c26
Sep 08 16:25:22.406712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:22.406682    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7hszn" event={"ID":"8f066f35-4ec8-4873-8fe5-e437340b85c1","Type":"ContainerStarted","Data":"904cb7dfbdc0ab0f3aa838a89d70cd1a5aedabcb1177ac208503522232f80c26"}
Sep 08 16:25:23.412724 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:23.412686    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7hszn" event={"ID":"8f066f35-4ec8-4873-8fe5-e437340b85c1","Type":"ContainerStarted","Data":"e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb"}
Sep 08 16:25:23.413155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:23.412772    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:23.431688 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:23.431642    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-7hszn" podStartSLOduration=1.370265473 podStartE2EDuration="2.431627768s" totalImagesPullingTime="1.061362295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:23.430607598 +0000 UTC m=+757.517271349" watchObservedRunningTime="2026-09-08 16:25:23.431627768 +0000 UTC m=+757.518291515"
Sep 08 16:25:34.418266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:34.418235    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:34.875564 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:34.875535    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7hszn"]
Sep 08 16:25:34.875726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:34.875705    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-7hszn" podUID="8f066f35-4ec8-4873-8fe5-e437340b85c1" containerName="limitador" containerID="cri-o://e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb" gracePeriod=30
Sep 08 16:25:35.426422 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.426398    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:35.466317 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.466241    2575 generic.go:356] "Generic (PLEG): container finished" podID="8f066f35-4ec8-4873-8fe5-e437340b85c1" containerID="e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb" exitCode=0
Sep 08 16:25:35.466317 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.466305    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-7hszn"
Sep 08 16:25:35.466463 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.466362    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7hszn" event={"ID":"8f066f35-4ec8-4873-8fe5-e437340b85c1","Type":"ContainerDied","Data":"e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb"}
Sep 08 16:25:35.466463 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.466435    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-7hszn" event={"ID":"8f066f35-4ec8-4873-8fe5-e437340b85c1","Type":"ContainerDied","Data":"904cb7dfbdc0ab0f3aa838a89d70cd1a5aedabcb1177ac208503522232f80c26"}
Sep 08 16:25:35.466528 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.466464    2575 scope.go:117] "RemoveContainer" containerID="e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb"
Sep 08 16:25:35.475230 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.475209    2575 scope.go:117] "RemoveContainer" containerID="e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb"
Sep 08 16:25:35.475486 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:25:35.475461    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb\": container with ID starting with e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb not found: ID does not exist" containerID="e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb"
Sep 08 16:25:35.475551 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.475495    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb"} err="failed to get container status \"e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb\": rpc error: code = NotFound desc = could not find container \"e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb\": container with ID starting with e8a6fb7b82d043f3027d2b80108b4792ec6d2e8e57e8fbadf021879843b0f5cb not found: ID does not exist"
Sep 08 16:25:35.559406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.559385    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f066f35-4ec8-4873-8fe5-e437340b85c1-config-file\") pod \"8f066f35-4ec8-4873-8fe5-e437340b85c1\" (UID: \"8f066f35-4ec8-4873-8fe5-e437340b85c1\") "
Sep 08 16:25:35.559520 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.559437    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b42d8\" (UniqueName: \"kubernetes.io/projected/8f066f35-4ec8-4873-8fe5-e437340b85c1-kube-api-access-b42d8\") pod \"8f066f35-4ec8-4873-8fe5-e437340b85c1\" (UID: \"8f066f35-4ec8-4873-8fe5-e437340b85c1\") "
Sep 08 16:25:35.559778 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.559755    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f066f35-4ec8-4873-8fe5-e437340b85c1-config-file" (OuterVolumeSpecName: "config-file") pod "8f066f35-4ec8-4873-8fe5-e437340b85c1" (UID: "8f066f35-4ec8-4873-8fe5-e437340b85c1"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:25:35.561377 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.561355    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f066f35-4ec8-4873-8fe5-e437340b85c1-kube-api-access-b42d8" (OuterVolumeSpecName: "kube-api-access-b42d8") pod "8f066f35-4ec8-4873-8fe5-e437340b85c1" (UID: "8f066f35-4ec8-4873-8fe5-e437340b85c1"). InnerVolumeSpecName "kube-api-access-b42d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:25:35.660687 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.660661    2575 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f066f35-4ec8-4873-8fe5-e437340b85c1-config-file\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:25:35.660806 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.660692    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b42d8\" (UniqueName: \"kubernetes.io/projected/8f066f35-4ec8-4873-8fe5-e437340b85c1-kube-api-access-b42d8\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:25:35.789524 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.789490    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7hszn"]
Sep 08 16:25:35.808080 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:35.808044    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-7hszn"]
Sep 08 16:25:36.505627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:36.505593    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f066f35-4ec8-4873-8fe5-e437340b85c1" path="/var/lib/kubelet/pods/8f066f35-4ec8-4873-8fe5-e437340b85c1/volumes"
Sep 08 16:25:39.502094 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.502063    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4z477"]
Sep 08 16:25:39.502553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.502413    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f066f35-4ec8-4873-8fe5-e437340b85c1" containerName="limitador"
Sep 08 16:25:39.502553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.502428    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f066f35-4ec8-4873-8fe5-e437340b85c1" containerName="limitador"
Sep 08 16:25:39.502553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.502496    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f066f35-4ec8-4873-8fe5-e437340b85c1" containerName="limitador"
Sep 08 16:25:39.504986 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.504967    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:39.507430 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.507409    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-5z29b\""
Sep 08 16:25:39.507571 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.507550    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 08 16:25:39.508387 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.508363    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 08 16:25:39.508387 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.508379    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 08 16:25:39.515237 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.515167    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4z477"]
Sep 08 16:25:39.591234 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.591207    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/066d862e-2178-4d32-8644-f629c14f1e0e-data\") pod \"postgres-69fb4594cf-4z477\" (UID: \"066d862e-2178-4d32-8644-f629c14f1e0e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:39.591340 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.591239    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r686\" (UniqueName: \"kubernetes.io/projected/066d862e-2178-4d32-8644-f629c14f1e0e-kube-api-access-8r686\") pod \"postgres-69fb4594cf-4z477\" (UID: \"066d862e-2178-4d32-8644-f629c14f1e0e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:39.692473 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.692443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/066d862e-2178-4d32-8644-f629c14f1e0e-data\") pod \"postgres-69fb4594cf-4z477\" (UID: \"066d862e-2178-4d32-8644-f629c14f1e0e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:39.692473 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.692471    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r686\" (UniqueName: \"kubernetes.io/projected/066d862e-2178-4d32-8644-f629c14f1e0e-kube-api-access-8r686\") pod \"postgres-69fb4594cf-4z477\" (UID: \"066d862e-2178-4d32-8644-f629c14f1e0e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:39.692809 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.692787    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/066d862e-2178-4d32-8644-f629c14f1e0e-data\") pod \"postgres-69fb4594cf-4z477\" (UID: \"066d862e-2178-4d32-8644-f629c14f1e0e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:39.702477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.702447    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r686\" (UniqueName: \"kubernetes.io/projected/066d862e-2178-4d32-8644-f629c14f1e0e-kube-api-access-8r686\") pod \"postgres-69fb4594cf-4z477\" (UID: \"066d862e-2178-4d32-8644-f629c14f1e0e\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:39.816532 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:39.816478    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:40.091061 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:40.091031    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4z477"]
Sep 08 16:25:40.094855 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:25:40.094824    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066d862e_2178_4d32_8644_f629c14f1e0e.slice/crio-3906b042d384fcbf584c5dd757322ad63e7a42a8c1936fbb71731930a8d51904 WatchSource:0}: Error finding container 3906b042d384fcbf584c5dd757322ad63e7a42a8c1936fbb71731930a8d51904: Status 404 returned error can't find the container with id 3906b042d384fcbf584c5dd757322ad63e7a42a8c1936fbb71731930a8d51904
Sep 08 16:25:40.491467 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:40.491385    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477" event={"ID":"066d862e-2178-4d32-8644-f629c14f1e0e","Type":"ContainerStarted","Data":"3906b042d384fcbf584c5dd757322ad63e7a42a8c1936fbb71731930a8d51904"}
Sep 08 16:25:45.523168 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:45.523135    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477" event={"ID":"066d862e-2178-4d32-8644-f629c14f1e0e","Type":"ContainerStarted","Data":"b4716c8c8b79ba2cc379154d2284dc5d920d75a9bb9e03238bb79041526ab221"}
Sep 08 16:25:45.523511 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:45.523234    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:25:45.539946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:45.539871    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477" podStartSLOduration=1.337886932 podStartE2EDuration="6.539851634s" totalImagesPullingTime="5.201964702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:25:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:25:45.538547023 +0000 UTC m=+779.625210795" watchObservedRunningTime="2026-09-08 16:25:45.539851634 +0000 UTC m=+779.626515382"
Sep 08 16:25:51.556989 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:25:51.556962    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4z477"
Sep 08 16:26:00.855732 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.855692    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57dc76869d-fctvw"]
Sep 08 16:26:00.875392 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.875361    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57dc76869d-fctvw"]
Sep 08 16:26:00.875553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.875502    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:00.878042 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.878013    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 08 16:26:00.878439 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.878013    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 08 16:26:00.878439 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.878194    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 08 16:26:00.880174 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.880152    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdv66\""
Sep 08 16:26:00.957046 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.957017    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:00.957046 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.957047    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmkrq\" (UniqueName: \"kubernetes.io/projected/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-kube-api-access-xmkrq\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:00.957219 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:00.957131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-webhook-cert\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:01.058059 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:01.058024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-webhook-cert\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:01.058223 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:01.058090    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:01.058223 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:01.058117    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmkrq\" (UniqueName: \"kubernetes.io/projected/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-kube-api-access-xmkrq\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:01.058354 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:26:01.058291    2575 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 08 16:26:01.058406 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:26:01.058378    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs podName:b1e0d425-0097-444b-bdf1-04e3b1d4efe1 nodeName:}" failed. No retries permitted until 2026-09-08 16:26:01.558356197 +0000 UTC m=+795.645019937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs") pod "maas-controller-57dc76869d-fctvw" (UID: "b1e0d425-0097-444b-bdf1-04e3b1d4efe1") : secret "maas-controller-metrics-tls" not found
Sep 08 16:26:01.060859 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:01.060837    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-webhook-cert\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:01.077637 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:01.077617    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmkrq\" (UniqueName: \"kubernetes.io/projected/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-kube-api-access-xmkrq\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:01.562332 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:01.562304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:01.564520 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:01.564501    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs\") pod \"maas-controller-57dc76869d-fctvw\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") " pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:01.789267 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:01.789228    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:02.068382 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:02.068358    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57dc76869d-fctvw"]
Sep 08 16:26:02.070346 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:26:02.070324    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1e0d425_0097_444b_bdf1_04e3b1d4efe1.slice/crio-d9b083ee7d5b76fa430fade62217ec42dac5304a9a0fd0ac9a0ac4dc6945fffc WatchSource:0}: Error finding container d9b083ee7d5b76fa430fade62217ec42dac5304a9a0fd0ac9a0ac4dc6945fffc: Status 404 returned error can't find the container with id d9b083ee7d5b76fa430fade62217ec42dac5304a9a0fd0ac9a0ac4dc6945fffc
Sep 08 16:26:02.599020 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:02.598981    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dc76869d-fctvw" event={"ID":"b1e0d425-0097-444b-bdf1-04e3b1d4efe1","Type":"ContainerStarted","Data":"d9b083ee7d5b76fa430fade62217ec42dac5304a9a0fd0ac9a0ac4dc6945fffc"}
Sep 08 16:26:05.617726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:05.617690    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dc76869d-fctvw" event={"ID":"b1e0d425-0097-444b-bdf1-04e3b1d4efe1","Type":"ContainerStarted","Data":"c3aa076442476d305fdb61fabd80f462bc649b6772e0c0174fda6adf2ca7c835"}
Sep 08 16:26:05.618131 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:05.617798    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:05.636343 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:05.636301    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-57dc76869d-fctvw" podStartSLOduration=3.154018767 podStartE2EDuration="5.636286204s" totalImagesPullingTime="2.482267437s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:26:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:26:05.635474623 +0000 UTC m=+799.722138454" watchObservedRunningTime="2026-09-08 16:26:05.636286204 +0000 UTC m=+799.722950223"
Sep 08 16:26:15.493897 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.493865    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57dc76869d-fctvw"]
Sep 08 16:26:15.494269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.494061    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-57dc76869d-fctvw" podUID="b1e0d425-0097-444b-bdf1-04e3b1d4efe1" containerName="manager" containerID="cri-o://c3aa076442476d305fdb61fabd80f462bc649b6772e0c0174fda6adf2ca7c835" gracePeriod=30
Sep 08 16:26:15.495709 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.495684    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:15.664869 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.664842    2575 generic.go:356] "Generic (PLEG): container finished" podID="b1e0d425-0097-444b-bdf1-04e3b1d4efe1" containerID="c3aa076442476d305fdb61fabd80f462bc649b6772e0c0174fda6adf2ca7c835" exitCode=0
Sep 08 16:26:15.664992 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.664900    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dc76869d-fctvw" event={"ID":"b1e0d425-0097-444b-bdf1-04e3b1d4efe1","Type":"ContainerDied","Data":"c3aa076442476d305fdb61fabd80f462bc649b6772e0c0174fda6adf2ca7c835"}
Sep 08 16:26:15.743569 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.743547    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:15.869282 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.869258    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-webhook-cert\") pod \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") "
Sep 08 16:26:15.869399 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.869295    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs\") pod \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") "
Sep 08 16:26:15.869399 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.869326    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xmkrq\" (UniqueName: \"kubernetes.io/projected/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-kube-api-access-xmkrq\") pod \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\" (UID: \"b1e0d425-0097-444b-bdf1-04e3b1d4efe1\") "
Sep 08 16:26:15.871364 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.871330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b1e0d425-0097-444b-bdf1-04e3b1d4efe1" (UID: "b1e0d425-0097-444b-bdf1-04e3b1d4efe1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:26:15.871482 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.871386    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b1e0d425-0097-444b-bdf1-04e3b1d4efe1" (UID: "b1e0d425-0097-444b-bdf1-04e3b1d4efe1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:26:15.871482 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.871405    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-kube-api-access-xmkrq" (OuterVolumeSpecName: "kube-api-access-xmkrq") pod "b1e0d425-0097-444b-bdf1-04e3b1d4efe1" (UID: "b1e0d425-0097-444b-bdf1-04e3b1d4efe1"). InnerVolumeSpecName "kube-api-access-xmkrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:26:15.969925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.969893    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmkrq\" (UniqueName: \"kubernetes.io/projected/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-kube-api-access-xmkrq\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:26:15.969925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.969923    2575 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-webhook-cert\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:26:15.969925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:15.969934    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b1e0d425-0097-444b-bdf1-04e3b1d4efe1-metrics-certs\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:26:16.670544 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:16.670515    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57dc76869d-fctvw" event={"ID":"b1e0d425-0097-444b-bdf1-04e3b1d4efe1","Type":"ContainerDied","Data":"d9b083ee7d5b76fa430fade62217ec42dac5304a9a0fd0ac9a0ac4dc6945fffc"}
Sep 08 16:26:16.670544 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:16.670532    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57dc76869d-fctvw"
Sep 08 16:26:16.671043 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:16.670573    2575 scope.go:117] "RemoveContainer" containerID="c3aa076442476d305fdb61fabd80f462bc649b6772e0c0174fda6adf2ca7c835"
Sep 08 16:26:16.690045 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:16.690021    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57dc76869d-fctvw"]
Sep 08 16:26:16.696444 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:16.696424    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57dc76869d-fctvw"]
Sep 08 16:26:18.506488 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:18.506451    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1e0d425-0097-444b-bdf1-04e3b1d4efe1" path="/var/lib/kubelet/pods/b1e0d425-0097-444b-bdf1-04e3b1d4efe1/volumes"
Sep 08 16:26:38.681671 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.681627    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"]
Sep 08 16:26:38.682218 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.682063    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1e0d425-0097-444b-bdf1-04e3b1d4efe1" containerName="manager"
Sep 08 16:26:38.682218 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.682081    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1e0d425-0097-444b-bdf1-04e3b1d4efe1" containerName="manager"
Sep 08 16:26:38.682218 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.682215    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1e0d425-0097-444b-bdf1-04e3b1d4efe1" containerName="manager"
Sep 08 16:26:38.688375 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.688356    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:38.696494 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.696474    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6dcdq\""
Sep 08 16:26:38.696772 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.696757    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 08 16:26:38.730647 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.730621    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"]
Sep 08 16:26:38.779214 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.779188    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"]
Sep 08 16:26:38.783045 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.783028    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:38.810659 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.810632    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"]
Sep 08 16:26:38.859186 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.859160    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztxt\" (UniqueName: \"kubernetes.io/projected/2a4a313a-9e47-4296-9b35-a07d2cc16aba-kube-api-access-pztxt\") pod \"payload-pre-processing-59fbc47bf8-9z79m\" (UID: \"2a4a313a-9e47-4296-9b35-a07d2cc16aba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:38.859302 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.859203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4a313a-9e47-4296-9b35-a07d2cc16aba-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9z79m\" (UID: \"2a4a313a-9e47-4296-9b35-a07d2cc16aba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:38.960111 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.960033    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcndj\" (UniqueName: \"kubernetes.io/projected/71def932-1f2f-44ac-a578-0e90c4bfdf63-kube-api-access-tcndj\") pod \"payload-processing-5b54b7fbb7-xsx6m\" (UID: \"71def932-1f2f-44ac-a578-0e90c4bfdf63\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:38.960224 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.960117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71def932-1f2f-44ac-a578-0e90c4bfdf63-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xsx6m\" (UID: \"71def932-1f2f-44ac-a578-0e90c4bfdf63\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:38.960224 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.960156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztxt\" (UniqueName: \"kubernetes.io/projected/2a4a313a-9e47-4296-9b35-a07d2cc16aba-kube-api-access-pztxt\") pod \"payload-pre-processing-59fbc47bf8-9z79m\" (UID: \"2a4a313a-9e47-4296-9b35-a07d2cc16aba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:38.960224 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.960196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4a313a-9e47-4296-9b35-a07d2cc16aba-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9z79m\" (UID: \"2a4a313a-9e47-4296-9b35-a07d2cc16aba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:38.960851 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.960827    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a4a313a-9e47-4296-9b35-a07d2cc16aba-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-9z79m\" (UID: \"2a4a313a-9e47-4296-9b35-a07d2cc16aba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:38.975942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.975917    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztxt\" (UniqueName: \"kubernetes.io/projected/2a4a313a-9e47-4296-9b35-a07d2cc16aba-kube-api-access-pztxt\") pod \"payload-pre-processing-59fbc47bf8-9z79m\" (UID: \"2a4a313a-9e47-4296-9b35-a07d2cc16aba\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:38.997840 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:38.997818    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:39.061061 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.061024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcndj\" (UniqueName: \"kubernetes.io/projected/71def932-1f2f-44ac-a578-0e90c4bfdf63-kube-api-access-tcndj\") pod \"payload-processing-5b54b7fbb7-xsx6m\" (UID: \"71def932-1f2f-44ac-a578-0e90c4bfdf63\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:39.061195 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.061112    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71def932-1f2f-44ac-a578-0e90c4bfdf63-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xsx6m\" (UID: \"71def932-1f2f-44ac-a578-0e90c4bfdf63\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:39.062269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.062179    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71def932-1f2f-44ac-a578-0e90c4bfdf63-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-xsx6m\" (UID: \"71def932-1f2f-44ac-a578-0e90c4bfdf63\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:39.072273 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.072234    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcndj\" (UniqueName: \"kubernetes.io/projected/71def932-1f2f-44ac-a578-0e90c4bfdf63-kube-api-access-tcndj\") pod \"payload-processing-5b54b7fbb7-xsx6m\" (UID: \"71def932-1f2f-44ac-a578-0e90c4bfdf63\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:39.092840 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.092818    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:39.286383 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.286359    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"]
Sep 08 16:26:39.289056 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:26:39.289029    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4a313a_9e47_4296_9b35_a07d2cc16aba.slice/crio-7b0bff663dee2478fae88f8fa0516f0208e2563131f4c8366d063490c41f2c32 WatchSource:0}: Error finding container 7b0bff663dee2478fae88f8fa0516f0208e2563131f4c8366d063490c41f2c32: Status 404 returned error can't find the container with id 7b0bff663dee2478fae88f8fa0516f0208e2563131f4c8366d063490c41f2c32
Sep 08 16:26:39.380013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.379985    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"]
Sep 08 16:26:39.382572 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:26:39.382549    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71def932_1f2f_44ac_a578_0e90c4bfdf63.slice/crio-28d985a89afe3f38ccf2332d62f815fd1e7c20bc68b9217da65a0880e436aa32 WatchSource:0}: Error finding container 28d985a89afe3f38ccf2332d62f815fd1e7c20bc68b9217da65a0880e436aa32: Status 404 returned error can't find the container with id 28d985a89afe3f38ccf2332d62f815fd1e7c20bc68b9217da65a0880e436aa32
Sep 08 16:26:39.782155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.782117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m" event={"ID":"2a4a313a-9e47-4296-9b35-a07d2cc16aba","Type":"ContainerStarted","Data":"7b0bff663dee2478fae88f8fa0516f0208e2563131f4c8366d063490c41f2c32"}
Sep 08 16:26:39.783116 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:39.783087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m" event={"ID":"71def932-1f2f-44ac-a578-0e90c4bfdf63","Type":"ContainerStarted","Data":"28d985a89afe3f38ccf2332d62f815fd1e7c20bc68b9217da65a0880e436aa32"}
Sep 08 16:26:41.800352 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:41.800318    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m" event={"ID":"71def932-1f2f-44ac-a578-0e90c4bfdf63","Type":"ContainerStarted","Data":"2ca1b76025620fc415642e68843e767f3019cf3cee5fd4b397e195b46ee4cd51"}
Sep 08 16:26:41.800810 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:41.800382    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:41.801749 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:41.801721    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m" event={"ID":"2a4a313a-9e47-4296-9b35-a07d2cc16aba","Type":"ContainerStarted","Data":"468d8c9021c474f933e5ec5999ee6a60b44b1d234c88c40bfe479f8fa15a0519"}
Sep 08 16:26:41.801878 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:41.801858    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:41.821855 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:41.821802    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m" podStartSLOduration=2.055905361 podStartE2EDuration="3.821784473s" totalImagesPullingTime="1.765879112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:26:41.820777594 +0000 UTC m=+835.907441353" watchObservedRunningTime="2026-09-08 16:26:41.821784473 +0000 UTC m=+835.908448224"
Sep 08 16:26:41.846198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:41.846151    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m" podStartSLOduration=1.979746011 podStartE2EDuration="3.846131143s" totalImagesPullingTime="1.866385132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:26:41.842732976 +0000 UTC m=+835.929396735" watchObservedRunningTime="2026-09-08 16:26:41.846131143 +0000 UTC m=+835.932794891"
Sep 08 16:26:52.808176 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:52.808141    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-xsx6m"
Sep 08 16:26:52.808702 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:52.808223    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-9z79m"
Sep 08 16:26:53.834087 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.834052    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"]
Sep 08 16:26:53.838720 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.838701    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.842570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.842552    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 08 16:26:53.843069 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.843055    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xc4pf\""
Sep 08 16:26:53.843238 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.843219    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 08 16:26:53.844324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.844307    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 08 16:26:53.862788 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.862761    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"]
Sep 08 16:26:53.878122 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.878096    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01aea10-5231-4441-9136-b45cca535a4e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.878214 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.878132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.878214 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.878149    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.878214 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.878180    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdjf\" (UniqueName: \"kubernetes.io/projected/f01aea10-5231-4441-9136-b45cca535a4e-kube-api-access-qpdjf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.878214 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.878209    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.878346 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.878223    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979083 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979056    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979194 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979194 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979135    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01aea10-5231-4441-9136-b45cca535a4e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979194 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979156    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979194 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdjf\" (UniqueName: \"kubernetes.io/projected/f01aea10-5231-4441-9136-b45cca535a4e-kube-api-access-qpdjf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979516 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979493    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979568 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979535    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.979568 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.979552    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.981358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.981340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f01aea10-5231-4441-9136-b45cca535a4e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.981629 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.981614    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f01aea10-5231-4441-9136-b45cca535a4e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:53.988927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:53.988907    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdjf\" (UniqueName: \"kubernetes.io/projected/f01aea10-5231-4441-9136-b45cca535a4e-kube-api-access-qpdjf\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj\" (UID: \"f01aea10-5231-4441-9136-b45cca535a4e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:54.148507 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:54.148443    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:26:54.437175 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:54.437094    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"]
Sep 08 16:26:54.440925 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:26:54.440895    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01aea10_5231_4441_9136_b45cca535a4e.slice/crio-d3be85566c1ddccb41c6e652b38f30c7381d7194542e78f8b4d7d600ba8080ef WatchSource:0}: Error finding container d3be85566c1ddccb41c6e652b38f30c7381d7194542e78f8b4d7d600ba8080ef: Status 404 returned error can't find the container with id d3be85566c1ddccb41c6e652b38f30c7381d7194542e78f8b4d7d600ba8080ef
Sep 08 16:26:54.442729 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:54.442706    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:26:54.861633 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:54.861571    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj" event={"ID":"f01aea10-5231-4441-9136-b45cca535a4e","Type":"ContainerStarted","Data":"d3be85566c1ddccb41c6e652b38f30c7381d7194542e78f8b4d7d600ba8080ef"}
Sep 08 16:26:55.867592 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:55.867544    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj" event={"ID":"f01aea10-5231-4441-9136-b45cca535a4e","Type":"ContainerStarted","Data":"7881f8a4ae108d9840a94fe516461035ad86d245ceeb87ee805ab08c3587f2ef"}
Sep 08 16:26:55.907661 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:26:55.907613    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj" podStartSLOduration=1.779019116 podStartE2EDuration="2.907600587s" totalImagesPullingTime="1.128581471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:26:55.904700452 +0000 UTC m=+849.991364201" watchObservedRunningTime="2026-09-08 16:26:55.907600587 +0000 UTC m=+849.994264335"
Sep 08 16:27:04.149470 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:04.149436    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:27:04.149939 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:04.149534    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:27:04.161971 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:04.161944    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:27:04.922091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:04.922056    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj"
Sep 08 16:27:29.867607 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.867556    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"]
Sep 08 16:27:29.870515 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.870496    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:29.877555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.877521    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 08 16:27:29.952515 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.952476    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:29.952515 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.952513    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:29.952746 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.952541    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdhz4\" (UniqueName: \"kubernetes.io/projected/713bc347-f48a-4330-8887-24d06cd4b18d-kube-api-access-pdhz4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:29.952746 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.952647    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:29.952746 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.952688    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:29.952746 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:29.952728    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/713bc347-f48a-4330-8887-24d06cd4b18d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.012862 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.012831    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"]
Sep 08 16:27:30.054032 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.053994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.054216 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.054049    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/713bc347-f48a-4330-8887-24d06cd4b18d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.054216 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.054094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.054216 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.054114    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.054216 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.054143    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdhz4\" (UniqueName: \"kubernetes.io/projected/713bc347-f48a-4330-8887-24d06cd4b18d-kube-api-access-pdhz4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.054216 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.054177    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.054477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.054421    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.054532 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.054467    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.054532 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.054508    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.056455 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.056435    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/713bc347-f48a-4330-8887-24d06cd4b18d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.056588 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.056560    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/713bc347-f48a-4330-8887-24d06cd4b18d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.068357 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.068332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdhz4\" (UniqueName: \"kubernetes.io/projected/713bc347-f48a-4330-8887-24d06cd4b18d-kube-api-access-pdhz4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j\" (UID: \"713bc347-f48a-4330-8887-24d06cd4b18d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.181174 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.181088    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:30.475437 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:30.475412    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"]
Sep 08 16:27:30.477916 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:27:30.477887    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713bc347_f48a_4330_8887_24d06cd4b18d.slice/crio-5c441553562b0defc2d659a52b64766295e5e92f8eec4978da1a0cfe7e329f74 WatchSource:0}: Error finding container 5c441553562b0defc2d659a52b64766295e5e92f8eec4978da1a0cfe7e329f74: Status 404 returned error can't find the container with id 5c441553562b0defc2d659a52b64766295e5e92f8eec4978da1a0cfe7e329f74
Sep 08 16:27:31.029154 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:31.029117    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j" event={"ID":"713bc347-f48a-4330-8887-24d06cd4b18d","Type":"ContainerStarted","Data":"b12cab82b76235a0b24db960bec03ca87f04331375dcb2777fdb1de6446e31e1"}
Sep 08 16:27:31.029668 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:31.029184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j" event={"ID":"713bc347-f48a-4330-8887-24d06cd4b18d","Type":"ContainerStarted","Data":"5c441553562b0defc2d659a52b64766295e5e92f8eec4978da1a0cfe7e329f74"}
Sep 08 16:27:38.362040 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.361988    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j" podStartSLOduration=9.200179915 podStartE2EDuration="9.361970521s" totalImagesPullingTime="161.790606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:31.096443941 +0000 UTC m=+885.183107688" watchObservedRunningTime="2026-09-08 16:27:38.361970521 +0000 UTC m=+892.448634268"
Sep 08 16:27:38.362704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.362686    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"]
Sep 08 16:27:38.365063 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.365046    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.371231 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.371209    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 08 16:27:38.404622 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.404593    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"]
Sep 08 16:27:38.428017 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.427990    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.428135 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.428061    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.428178 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.428146    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgtlr\" (UniqueName: \"kubernetes.io/projected/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-kube-api-access-kgtlr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.428178 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.428168    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.428291 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.428212    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.428291 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.428264    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.529498 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.529470    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.529714 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.529529    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.529714 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.529565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.529714 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.529648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.529714 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.529688    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.529912 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.529747    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgtlr\" (UniqueName: \"kubernetes.io/projected/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-kube-api-access-kgtlr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.530570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.530484    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.530570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.530502    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.530754 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.530698    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.532787 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.532763    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.532895 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.532870    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.541974 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.541953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgtlr\" (UniqueName: \"kubernetes.io/projected/4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6-kube-api-access-kgtlr\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-4lm2n\" (UID: \"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.697645 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.697522    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:38.984458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:38.984377    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"]
Sep 08 16:27:38.987543 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:27:38.987511    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a2138c3_c9d7_41f6_a3ed_24eaa0e05ff6.slice/crio-b1fd5361b9e18a95f3643a27dcce606043fd677cb7dc7c97023dc3d4fb49a694 WatchSource:0}: Error finding container b1fd5361b9e18a95f3643a27dcce606043fd677cb7dc7c97023dc3d4fb49a694: Status 404 returned error can't find the container with id b1fd5361b9e18a95f3643a27dcce606043fd677cb7dc7c97023dc3d4fb49a694
Sep 08 16:27:39.068648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:39.068605    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n" event={"ID":"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6","Type":"ContainerStarted","Data":"b1fd5361b9e18a95f3643a27dcce606043fd677cb7dc7c97023dc3d4fb49a694"}
Sep 08 16:27:40.075344 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:40.075311    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n" event={"ID":"4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6","Type":"ContainerStarted","Data":"0b1fc5e474fd1583e97f94f9611c3c35aeb405c2d07fbec3e98e24e96036578c"}
Sep 08 16:27:40.104372 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:40.104310    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n" podStartSLOduration=1.927215865 podStartE2EDuration="2.104290501s" totalImagesPullingTime="177.074636ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:27:40.101738054 +0000 UTC m=+894.188401799" watchObservedRunningTime="2026-09-08 16:27:40.104290501 +0000 UTC m=+894.190954248"
Sep 08 16:27:40.181549 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:40.181506    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:40.181766 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:40.181604    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:40.194761 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:40.194730    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:41.092438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:41.092406    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j"
Sep 08 16:27:46.456963 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:46.456935    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:27:46.457762 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:46.457738    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:27:46.462551 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:46.462534    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:27:46.463367 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:46.463350    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:27:48.698199 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:48.698161    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:48.698631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:48.698258    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:48.711091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:48.711070    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:27:49.142215 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:27:49.142188    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-4lm2n"
Sep 08 16:29:32.358389 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.358352    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"]
Sep 08 16:29:32.361069 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.361051    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:32.364799 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.364766    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0d9aa4\""
Sep 08 16:29:32.364799 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.364792    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0d9aa4-dockercfg-stk6v\""
Sep 08 16:29:32.391816 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.391770    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"]
Sep 08 16:29:32.431857 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.431828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qxqz\" (UniqueName: \"kubernetes.io/projected/0259c31e-91c2-4573-bc93-bcc747364acb-kube-api-access-8qxqz\") pod \"payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k\" (UID: \"0259c31e-91c2-4573-bc93-bcc747364acb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:32.431967 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.431867    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0259c31e-91c2-4573-bc93-bcc747364acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k\" (UID: \"0259c31e-91c2-4573-bc93-bcc747364acb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:32.435150 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.435128    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"]
Sep 08 16:29:32.437512 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.437495    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:32.458895 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.458872    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"]
Sep 08 16:29:32.532794 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.532766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a690817-566b-42e9-8d07-8a8691d245d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml\" (UID: \"2a690817-566b-42e9-8d07-8a8691d245d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:32.532946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.532830    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qxqz\" (UniqueName: \"kubernetes.io/projected/0259c31e-91c2-4573-bc93-bcc747364acb-kube-api-access-8qxqz\") pod \"payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k\" (UID: \"0259c31e-91c2-4573-bc93-bcc747364acb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:32.532946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.532896    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0259c31e-91c2-4573-bc93-bcc747364acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k\" (UID: \"0259c31e-91c2-4573-bc93-bcc747364acb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:32.532946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.532937    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwpqf\" (UniqueName: \"kubernetes.io/projected/2a690817-566b-42e9-8d07-8a8691d245d5-kube-api-access-pwpqf\") pod \"payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml\" (UID: \"2a690817-566b-42e9-8d07-8a8691d245d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:32.533545 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.533527    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0259c31e-91c2-4573-bc93-bcc747364acb-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k\" (UID: \"0259c31e-91c2-4573-bc93-bcc747364acb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:32.544467 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.544444    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qxqz\" (UniqueName: \"kubernetes.io/projected/0259c31e-91c2-4573-bc93-bcc747364acb-kube-api-access-8qxqz\") pod \"payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k\" (UID: \"0259c31e-91c2-4573-bc93-bcc747364acb\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:32.634033 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.633955    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwpqf\" (UniqueName: \"kubernetes.io/projected/2a690817-566b-42e9-8d07-8a8691d245d5-kube-api-access-pwpqf\") pod \"payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml\" (UID: \"2a690817-566b-42e9-8d07-8a8691d245d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:32.634146 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.634113    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a690817-566b-42e9-8d07-8a8691d245d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml\" (UID: \"2a690817-566b-42e9-8d07-8a8691d245d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:32.634787 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.634771    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a690817-566b-42e9-8d07-8a8691d245d5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml\" (UID: \"2a690817-566b-42e9-8d07-8a8691d245d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:32.643867 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.643846    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwpqf\" (UniqueName: \"kubernetes.io/projected/2a690817-566b-42e9-8d07-8a8691d245d5-kube-api-access-pwpqf\") pod \"payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml\" (UID: \"2a690817-566b-42e9-8d07-8a8691d245d5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:32.671133 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.671112    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:32.748624 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.748355    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:32.950731 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:32.950709    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"]
Sep 08 16:29:32.952152 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:29:32.952127    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0259c31e_91c2_4573_bc93_bcc747364acb.slice/crio-460a9762ba70a5a10c1fb708a431c07652207adcfbfbc901a0d506be9bb3f895 WatchSource:0}: Error finding container 460a9762ba70a5a10c1fb708a431c07652207adcfbfbc901a0d506be9bb3f895: Status 404 returned error can't find the container with id 460a9762ba70a5a10c1fb708a431c07652207adcfbfbc901a0d506be9bb3f895
Sep 08 16:29:33.030813 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.030781    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"]
Sep 08 16:29:33.031796 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:29:33.031773    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a690817_566b_42e9_8d07_8a8691d245d5.slice/crio-468c13fd5646d14d8a67a5cb9451fce57966d8dadbad08904eb2764cd737179f WatchSource:0}: Error finding container 468c13fd5646d14d8a67a5cb9451fce57966d8dadbad08904eb2764cd737179f: Status 404 returned error can't find the container with id 468c13fd5646d14d8a67a5cb9451fce57966d8dadbad08904eb2764cd737179f
Sep 08 16:29:33.636069 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.636034    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml" event={"ID":"2a690817-566b-42e9-8d07-8a8691d245d5","Type":"ContainerStarted","Data":"500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee"}
Sep 08 16:29:33.636507 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.636096    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml" event={"ID":"2a690817-566b-42e9-8d07-8a8691d245d5","Type":"ContainerStarted","Data":"468c13fd5646d14d8a67a5cb9451fce57966d8dadbad08904eb2764cd737179f"}
Sep 08 16:29:33.636507 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.636148    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:33.637812 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.637785    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k" event={"ID":"0259c31e-91c2-4573-bc93-bcc747364acb","Type":"ContainerStarted","Data":"113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f"}
Sep 08 16:29:33.637940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.637833    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k" event={"ID":"0259c31e-91c2-4573-bc93-bcc747364acb","Type":"ContainerStarted","Data":"460a9762ba70a5a10c1fb708a431c07652207adcfbfbc901a0d506be9bb3f895"}
Sep 08 16:29:33.637940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.637854    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:33.660353 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.660315    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml" podStartSLOduration=1.6603021070000001 podStartE2EDuration="1.660302107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:33.65856703 +0000 UTC m=+1007.745230777" watchObservedRunningTime="2026-09-08 16:29:33.660302107 +0000 UTC m=+1007.746965858"
Sep 08 16:29:33.685326 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:33.685287    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k" podStartSLOduration=1.68527451 podStartE2EDuration="1.68527451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:33.683667866 +0000 UTC m=+1007.770331614" watchObservedRunningTime="2026-09-08 16:29:33.68527451 +0000 UTC m=+1007.771938259"
Sep 08 16:29:40.328830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.328795    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"]
Sep 08 16:29:40.331346 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.331328    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:40.334361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.334337    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ec338109-dockercfg-zgbc5\""
Sep 08 16:29:40.334530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.334509    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ec338109\""
Sep 08 16:29:40.367283 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.367261    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"]
Sep 08 16:29:40.369424 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.369410    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:40.390698 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.390674    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"]
Sep 08 16:29:40.404201 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.404182    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95lk4\" (UniqueName: \"kubernetes.io/projected/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-kube-api-access-95lk4\") pod \"payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59\" (UID: \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\") " pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:40.404302 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.404225    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c1bdc-868c-4b0a-ac44-2c245389f83d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r\" (UID: \"749c1bdc-868c-4b0a-ac44-2c245389f83d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:40.404380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.404352    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6nq\" (UniqueName: \"kubernetes.io/projected/749c1bdc-868c-4b0a-ac44-2c245389f83d-kube-api-access-sh6nq\") pod \"payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r\" (UID: \"749c1bdc-868c-4b0a-ac44-2c245389f83d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:40.404441 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.404398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59\" (UID: \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\") " pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:40.409549 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.409529    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"]
Sep 08 16:29:40.505984 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.505224    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6nq\" (UniqueName: \"kubernetes.io/projected/749c1bdc-868c-4b0a-ac44-2c245389f83d-kube-api-access-sh6nq\") pod \"payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r\" (UID: \"749c1bdc-868c-4b0a-ac44-2c245389f83d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:40.505984 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.505262    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59\" (UID: \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\") " pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:40.505984 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.505304    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95lk4\" (UniqueName: \"kubernetes.io/projected/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-kube-api-access-95lk4\") pod \"payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59\" (UID: \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\") " pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:40.505984 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.505352    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c1bdc-868c-4b0a-ac44-2c245389f83d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r\" (UID: \"749c1bdc-868c-4b0a-ac44-2c245389f83d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:40.506714 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.506690    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c1bdc-868c-4b0a-ac44-2c245389f83d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r\" (UID: \"749c1bdc-868c-4b0a-ac44-2c245389f83d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:40.507598 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.507549    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-plugins-config-volume\") pod \"payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59\" (UID: \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\") " pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:40.522026 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.521996    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6nq\" (UniqueName: \"kubernetes.io/projected/749c1bdc-868c-4b0a-ac44-2c245389f83d-kube-api-access-sh6nq\") pod \"payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r\" (UID: \"749c1bdc-868c-4b0a-ac44-2c245389f83d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:40.522698 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.522681    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95lk4\" (UniqueName: \"kubernetes.io/projected/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-kube-api-access-95lk4\") pod \"payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59\" (UID: \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\") " pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:40.642744 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.642677    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:40.678448 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.678421    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:40.928808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.928782    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"]
Sep 08 16:29:40.929838 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:29:40.929811    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749c1bdc_868c_4b0a_ac44_2c245389f83d.slice/crio-6f435898183543a479641f001b681e454eb6e6efaf880cdf0a2a7359d266b1df WatchSource:0}: Error finding container 6f435898183543a479641f001b681e454eb6e6efaf880cdf0a2a7359d266b1df: Status 404 returned error can't find the container with id 6f435898183543a479641f001b681e454eb6e6efaf880cdf0a2a7359d266b1df
Sep 08 16:29:40.986427 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:40.986405    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"]
Sep 08 16:29:40.988447 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:29:40.988419    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9edc65_e3dd_48d6_95d2_3bff0a739fb2.slice/crio-5a446a61ceb7d55f4c09de430c7964511c227d6ef8de5cc70a65167e00dd880c WatchSource:0}: Error finding container 5a446a61ceb7d55f4c09de430c7964511c227d6ef8de5cc70a65167e00dd880c: Status 404 returned error can't find the container with id 5a446a61ceb7d55f4c09de430c7964511c227d6ef8de5cc70a65167e00dd880c
Sep 08 16:29:41.681484 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.681441    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59" event={"ID":"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2","Type":"ContainerStarted","Data":"00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a"}
Sep 08 16:29:41.681909 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.681554    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:41.681909 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.681613    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59" event={"ID":"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2","Type":"ContainerStarted","Data":"5a446a61ceb7d55f4c09de430c7964511c227d6ef8de5cc70a65167e00dd880c"}
Sep 08 16:29:41.683045 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.683019    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r" event={"ID":"749c1bdc-868c-4b0a-ac44-2c245389f83d","Type":"ContainerStarted","Data":"5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba"}
Sep 08 16:29:41.683152 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.683077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:41.683152 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.683096    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r" event={"ID":"749c1bdc-868c-4b0a-ac44-2c245389f83d","Type":"ContainerStarted","Data":"6f435898183543a479641f001b681e454eb6e6efaf880cdf0a2a7359d266b1df"}
Sep 08 16:29:41.728991 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.728946    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59" podStartSLOduration=1.7289349349999998 podStartE2EDuration="1.728934935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:41.72784521 +0000 UTC m=+1015.814508961" watchObservedRunningTime="2026-09-08 16:29:41.728934935 +0000 UTC m=+1015.815598683"
Sep 08 16:29:41.794644 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.794564    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r" podStartSLOduration=1.794545643 podStartE2EDuration="1.794545643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:41.793357679 +0000 UTC m=+1015.880021428" watchObservedRunningTime="2026-09-08 16:29:41.794545643 +0000 UTC m=+1015.881209392"
Sep 08 16:29:41.944928 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.944848    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"]
Sep 08 16:29:41.947261 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.947238    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:41.951002 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.950976    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-293166-dockercfg-2jkql\""
Sep 08 16:29:41.951114 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.951046    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-293166\""
Sep 08 16:29:41.966246 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:41.966218    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"]
Sep 08 16:29:42.019684 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.019619    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e148e8-b6f5-44af-8a7d-c7324f5159d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x\" (UID: \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:42.019832 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.019711    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmnrb\" (UniqueName: \"kubernetes.io/projected/99e148e8-b6f5-44af-8a7d-c7324f5159d3-kube-api-access-pmnrb\") pod \"payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x\" (UID: \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:42.120694 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.120656    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e148e8-b6f5-44af-8a7d-c7324f5159d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x\" (UID: \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:42.120887 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.120710    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmnrb\" (UniqueName: \"kubernetes.io/projected/99e148e8-b6f5-44af-8a7d-c7324f5159d3-kube-api-access-pmnrb\") pod \"payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x\" (UID: \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:42.121643 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.121603    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e148e8-b6f5-44af-8a7d-c7324f5159d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x\" (UID: \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:42.134436 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.134406    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmnrb\" (UniqueName: \"kubernetes.io/projected/99e148e8-b6f5-44af-8a7d-c7324f5159d3-kube-api-access-pmnrb\") pod \"payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x\" (UID: \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:42.257820 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.257742    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:42.541409 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.541376    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"]
Sep 08 16:29:42.542662 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:29:42.542631    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e148e8_b6f5_44af_8a7d_c7324f5159d3.slice/crio-468b0b81be2b31798279979f062f78269723ac83d2357c583a3fb8a84ad370b9 WatchSource:0}: Error finding container 468b0b81be2b31798279979f062f78269723ac83d2357c583a3fb8a84ad370b9: Status 404 returned error can't find the container with id 468b0b81be2b31798279979f062f78269723ac83d2357c583a3fb8a84ad370b9
Sep 08 16:29:42.691650 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.691612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x" event={"ID":"99e148e8-b6f5-44af-8a7d-c7324f5159d3","Type":"ContainerStarted","Data":"309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c"}
Sep 08 16:29:42.692091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.691787    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:29:42.692091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.691847    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x" event={"ID":"99e148e8-b6f5-44af-8a7d-c7324f5159d3","Type":"ContainerStarted","Data":"468b0b81be2b31798279979f062f78269723ac83d2357c583a3fb8a84ad370b9"}
Sep 08 16:29:42.718506 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:42.718429    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x" podStartSLOduration=1.718404916 podStartE2EDuration="1.718404916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:29:42.716686309 +0000 UTC m=+1016.803350058" watchObservedRunningTime="2026-09-08 16:29:42.718404916 +0000 UTC m=+1016.805068664"
Sep 08 16:29:44.646264 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:44.646233    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:29:44.646722 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:44.646351    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:29:52.693651 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:52.693619    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:29:52.694038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:52.693694    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:29:53.699478 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:29:53.699450    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:30:00.149165 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.149131    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"]
Sep 08 16:30:00.152548 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.152526    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx"]
Sep 08 16:30:00.152696 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.152666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"
Sep 08 16:30:00.155810 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.155791    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:30:00.155902 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.155884    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"]
Sep 08 16:30:00.155980 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.155962    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx"
Sep 08 16:30:00.158728 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.158709    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"
Sep 08 16:30:00.164734 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.164714    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"
Sep 08 16:30:00.174218 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.174195    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx"]
Sep 08 16:30:00.177778 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.177726    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"]
Sep 08 16:30:00.180067 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.179210    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx"
Sep 08 16:30:00.181801 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.181538    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"]
Sep 08 16:30:00.226529 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.226415    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"
Sep 08 16:30:00.490442 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.490404    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"]
Sep 08 16:30:00.492385 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:30:00.492338    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd981481e_28b7_4303_a29e_86314a4f97f6.slice/crio-1192959e50c134802682b000468b6f695bd463150279f2d52baa87b196daada7 WatchSource:0}: Error finding container 1192959e50c134802682b000468b6f695bd463150279f2d52baa87b196daada7: Status 404 returned error can't find the container with id 1192959e50c134802682b000468b6f695bd463150279f2d52baa87b196daada7
Sep 08 16:30:00.495139 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.495110    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx"]
Sep 08 16:30:00.525042 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.525021    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"]
Sep 08 16:30:00.525748 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:30:00.525725    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164442cb_e9fe_493a_a38e_1b6b98c74974.slice/crio-f21f3a4b9bdf2366db17a2b47ff514e95ff42c6f8740ab3158643a7d278d2e87 WatchSource:0}: Error finding container f21f3a4b9bdf2366db17a2b47ff514e95ff42c6f8740ab3158643a7d278d2e87: Status 404 returned error can't find the container with id f21f3a4b9bdf2366db17a2b47ff514e95ff42c6f8740ab3158643a7d278d2e87
Sep 08 16:30:00.784802 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.784720    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79" event={"ID":"d981481e-28b7-4303-a29e-86314a4f97f6","Type":"ContainerStarted","Data":"1192959e50c134802682b000468b6f695bd463150279f2d52baa87b196daada7"}
Sep 08 16:30:00.786009 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.785987    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx" event={"ID":"cd8c51df-0194-4b91-b0f3-3e521726bf87","Type":"ContainerStarted","Data":"f77dfdb1c4675c845cf833820c6c8e12943bcbc4e76a65f5e788f8c132179dfb"}
Sep 08 16:30:00.787213 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:00.787186    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw" event={"ID":"164442cb-e9fe-493a-a38e-1b6b98c74974","Type":"ContainerStarted","Data":"f21f3a4b9bdf2366db17a2b47ff514e95ff42c6f8740ab3158643a7d278d2e87"}
Sep 08 16:30:01.110662 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.110626    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"]
Sep 08 16:30:01.113681 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.113663    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.119918 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.119890    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b41034\""
Sep 08 16:30:01.120047 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.119984    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b41034-dockercfg-7mng2\""
Sep 08 16:30:01.134297 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.134271    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"]
Sep 08 16:30:01.177762 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.177734    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"]
Sep 08 16:30:01.180616 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.180574    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:01.198747 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.198721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8drqd\" (UniqueName: \"kubernetes.io/projected/7f387416-57b6-47eb-996d-d6a439e36d49-kube-api-access-8drqd\") pod \"payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2\" (UID: \"7f387416-57b6-47eb-996d-d6a439e36d49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.198863 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.198788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f387416-57b6-47eb-996d-d6a439e36d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2\" (UID: \"7f387416-57b6-47eb-996d-d6a439e36d49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.204511 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.204488    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"]
Sep 08 16:30:01.300227 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.300196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8drqd\" (UniqueName: \"kubernetes.io/projected/7f387416-57b6-47eb-996d-d6a439e36d49-kube-api-access-8drqd\") pod \"payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2\" (UID: \"7f387416-57b6-47eb-996d-d6a439e36d49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.300355 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.300251    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e88599-2cd9-422e-a851-6b82a1ddbe73-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz\" (UID: \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:01.300355 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.300280    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f387416-57b6-47eb-996d-d6a439e36d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2\" (UID: \"7f387416-57b6-47eb-996d-d6a439e36d49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.300355 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.300336    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmpzn\" (UniqueName: \"kubernetes.io/projected/f7e88599-2cd9-422e-a851-6b82a1ddbe73-kube-api-access-cmpzn\") pod \"payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz\" (UID: \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:01.300971 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.300953    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f387416-57b6-47eb-996d-d6a439e36d49-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2\" (UID: \"7f387416-57b6-47eb-996d-d6a439e36d49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.322467 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.322432    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8drqd\" (UniqueName: \"kubernetes.io/projected/7f387416-57b6-47eb-996d-d6a439e36d49-kube-api-access-8drqd\") pod \"payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2\" (UID: \"7f387416-57b6-47eb-996d-d6a439e36d49\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.401278 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.401202    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e88599-2cd9-422e-a851-6b82a1ddbe73-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz\" (UID: \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:01.401278 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.401248    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmpzn\" (UniqueName: \"kubernetes.io/projected/f7e88599-2cd9-422e-a851-6b82a1ddbe73-kube-api-access-cmpzn\") pod \"payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz\" (UID: \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:01.401942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.401921    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e88599-2cd9-422e-a851-6b82a1ddbe73-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz\" (UID: \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:01.412252 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.412224    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmpzn\" (UniqueName: \"kubernetes.io/projected/f7e88599-2cd9-422e-a851-6b82a1ddbe73-kube-api-access-cmpzn\") pod \"payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz\" (UID: \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:01.427090 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.427059    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.499663 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.499609    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:01.701733 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.701649    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"]
Sep 08 16:30:01.702490 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:30:01.702391    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f387416_57b6_47eb_996d_d6a439e36d49.slice/crio-06b8a338bc21b0a2408ed95e9fb5870b5f3ce088cd3d7056c9d2a2e634bebe27 WatchSource:0}: Error finding container 06b8a338bc21b0a2408ed95e9fb5870b5f3ce088cd3d7056c9d2a2e634bebe27: Status 404 returned error can't find the container with id 06b8a338bc21b0a2408ed95e9fb5870b5f3ce088cd3d7056c9d2a2e634bebe27
Sep 08 16:30:01.801714 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.801685    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"]
Sep 08 16:30:01.804369 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:30:01.803989    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e88599_2cd9_422e_a851_6b82a1ddbe73.slice/crio-e7a2cbffeb2954abf2c0288f50d6f0a8ce231b5ef7e9e30dab6cd0ceba7b3573 WatchSource:0}: Error finding container e7a2cbffeb2954abf2c0288f50d6f0a8ce231b5ef7e9e30dab6cd0ceba7b3573: Status 404 returned error can't find the container with id e7a2cbffeb2954abf2c0288f50d6f0a8ce231b5ef7e9e30dab6cd0ceba7b3573
Sep 08 16:30:01.804369 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.804217    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2" event={"ID":"7f387416-57b6-47eb-996d-d6a439e36d49","Type":"ContainerStarted","Data":"8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88"}
Sep 08 16:30:01.804369 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.804274    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2" event={"ID":"7f387416-57b6-47eb-996d-d6a439e36d49","Type":"ContainerStarted","Data":"06b8a338bc21b0a2408ed95e9fb5870b5f3ce088cd3d7056c9d2a2e634bebe27"}
Sep 08 16:30:01.804369 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.804326    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:01.834856 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:01.833771    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2" podStartSLOduration=0.833749695 podStartE2EDuration="833.749695ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:01.833079661 +0000 UTC m=+1035.919743410" watchObservedRunningTime="2026-09-08 16:30:01.833749695 +0000 UTC m=+1035.920413443"
Sep 08 16:30:02.812957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:02.812854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz" event={"ID":"f7e88599-2cd9-422e-a851-6b82a1ddbe73","Type":"ContainerStarted","Data":"7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641"}
Sep 08 16:30:02.812957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:02.812918    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz" event={"ID":"f7e88599-2cd9-422e-a851-6b82a1ddbe73","Type":"ContainerStarted","Data":"e7a2cbffeb2954abf2c0288f50d6f0a8ce231b5ef7e9e30dab6cd0ceba7b3573"}
Sep 08 16:30:02.853685 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:02.853621    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz" podStartSLOduration=1.853598353 podStartE2EDuration="1.853598353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:02.846361859 +0000 UTC m=+1036.933025608" watchObservedRunningTime="2026-09-08 16:30:02.853598353 +0000 UTC m=+1036.940262104"
Sep 08 16:30:03.825021 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:03.824985    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79" event={"ID":"d981481e-28b7-4303-a29e-86314a4f97f6","Type":"ContainerStarted","Data":"149eea2f43e72a5abd3fec301a4d2dee9f1034d9b4e38cb4320cb9f0a7a6d4f8"}
Sep 08 16:30:03.825468 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:03.825053    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:03.840487 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:03.840396    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79" podStartSLOduration=0.88737368 podStartE2EDuration="3.840372204s" totalImagesPullingTime="2.952998524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:03.839570304 +0000 UTC m=+1037.926234065" watchObservedRunningTime="2026-09-08 16:30:03.840372204 +0000 UTC m=+1037.927035952"
Sep 08 16:30:04.832349 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:04.832315    2575 generic.go:356] "Generic (PLEG): container finished" podID="d981481e-28b7-4303-a29e-86314a4f97f6" containerID="149eea2f43e72a5abd3fec301a4d2dee9f1034d9b4e38cb4320cb9f0a7a6d4f8" exitCode=0
Sep 08 16:30:04.832805 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:04.832403    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79" event={"ID":"d981481e-28b7-4303-a29e-86314a4f97f6","Type":"ContainerDied","Data":"149eea2f43e72a5abd3fec301a4d2dee9f1034d9b4e38cb4320cb9f0a7a6d4f8"}
Sep 08 16:30:04.834070 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:04.834044    2575 generic.go:356] "Generic (PLEG): container finished" podID="cd8c51df-0194-4b91-b0f3-3e521726bf87" containerID="43dbb69331a1c4f1d4bd82c5173ff86ec5f16bf633e1766b421f0a5b5879dd8e" exitCode=0
Sep 08 16:30:04.834186 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:04.834118    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx" event={"ID":"cd8c51df-0194-4b91-b0f3-3e521726bf87","Type":"ContainerDied","Data":"43dbb69331a1c4f1d4bd82c5173ff86ec5f16bf633e1766b421f0a5b5879dd8e"}
Sep 08 16:30:04.835866 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:04.835835    2575 generic.go:356] "Generic (PLEG): container finished" podID="164442cb-e9fe-493a-a38e-1b6b98c74974" containerID="19fb0bd898e0e17e42ab77f917f942533b5f71f18d130c3c957cbfdd2f5e3848" exitCode=0
Sep 08 16:30:04.835966 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:04.835881    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw" event={"ID":"164442cb-e9fe-493a-a38e-1b6b98c74974","Type":"ContainerDied","Data":"19fb0bd898e0e17e42ab77f917f942533b5f71f18d130c3c957cbfdd2f5e3848"}
Sep 08 16:30:05.983285 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:05.983254    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx"
Sep 08 16:30:06.028530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.028500    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"
Sep 08 16:30:06.067611 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.067574    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"
Sep 08 16:30:06.848193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.848159    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"
Sep 08 16:30:06.848365 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.848163    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79" event={"ID":"d981481e-28b7-4303-a29e-86314a4f97f6","Type":"ContainerDied","Data":"1192959e50c134802682b000468b6f695bd463150279f2d52baa87b196daada7"}
Sep 08 16:30:06.848365 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.848309    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1192959e50c134802682b000468b6f695bd463150279f2d52baa87b196daada7"
Sep 08 16:30:06.849570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.849550    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx"
Sep 08 16:30:06.849730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.849601    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29814750-rq8jx" event={"ID":"cd8c51df-0194-4b91-b0f3-3e521726bf87","Type":"ContainerDied","Data":"f77dfdb1c4675c845cf833820c6c8e12943bcbc4e76a65f5e788f8c132179dfb"}
Sep 08 16:30:06.849730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.849647    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f77dfdb1c4675c845cf833820c6c8e12943bcbc4e76a65f5e788f8c132179dfb"
Sep 08 16:30:06.851001 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.850974    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw" event={"ID":"164442cb-e9fe-493a-a38e-1b6b98c74974","Type":"ContainerDied","Data":"f21f3a4b9bdf2366db17a2b47ff514e95ff42c6f8740ab3158643a7d278d2e87"}
Sep 08 16:30:06.851001 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.850989    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"
Sep 08 16:30:06.851136 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:06.851009    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f21f3a4b9bdf2366db17a2b47ff514e95ff42c6f8740ab3158643a7d278d2e87"
Sep 08 16:30:07.985878 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.985842    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"]
Sep 08 16:30:07.986501 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986478    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d981481e-28b7-4303-a29e-86314a4f97f6" containerName="cleanup"
Sep 08 16:30:07.986617 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986504    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d981481e-28b7-4303-a29e-86314a4f97f6" containerName="cleanup"
Sep 08 16:30:07.986617 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986521    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="164442cb-e9fe-493a-a38e-1b6b98c74974" containerName="cleanup"
Sep 08 16:30:07.986617 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986529    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="164442cb-e9fe-493a-a38e-1b6b98c74974" containerName="cleanup"
Sep 08 16:30:07.986617 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986543    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8c51df-0194-4b91-b0f3-3e521726bf87" containerName="cleanup"
Sep 08 16:30:07.986617 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986552    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8c51df-0194-4b91-b0f3-3e521726bf87" containerName="cleanup"
Sep 08 16:30:07.986868 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986667    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8c51df-0194-4b91-b0f3-3e521726bf87" containerName="cleanup"
Sep 08 16:30:07.986868 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986683    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="164442cb-e9fe-493a-a38e-1b6b98c74974" containerName="cleanup"
Sep 08 16:30:07.986868 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.986696    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d981481e-28b7-4303-a29e-86314a4f97f6" containerName="cleanup"
Sep 08 16:30:07.994087 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.994021    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"
Sep 08 16:30:07.997049 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.997020    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:30:07.998547 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:07.998517    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"]
Sep 08 16:30:08.169670 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:08.169633    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5c5023-9ed0-4471-865d-4f52eb8129ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec338109-2f7lc\" (UID: \"0e5c5023-9ed0-4471-865d-4f52eb8129ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"
Sep 08 16:30:08.270748 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:08.270669    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5c5023-9ed0-4471-865d-4f52eb8129ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec338109-2f7lc\" (UID: \"0e5c5023-9ed0-4471-865d-4f52eb8129ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"
Sep 08 16:30:08.271326 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:08.271304    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5c5023-9ed0-4471-865d-4f52eb8129ec-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ec338109-2f7lc\" (UID: \"0e5c5023-9ed0-4471-865d-4f52eb8129ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"
Sep 08 16:30:08.309633 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:08.309605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"
Sep 08 16:30:08.581102 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:08.581076    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"]
Sep 08 16:30:08.583889 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:30:08.583847    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5c5023_9ed0_4471_865d_4f52eb8129ec.slice/crio-da98c21579af4bc0d9eabc1bbedd703b1bb0eb26b25fd11314f2bfb025a885a6 WatchSource:0}: Error finding container da98c21579af4bc0d9eabc1bbedd703b1bb0eb26b25fd11314f2bfb025a885a6: Status 404 returned error can't find the container with id da98c21579af4bc0d9eabc1bbedd703b1bb0eb26b25fd11314f2bfb025a885a6
Sep 08 16:30:08.862402 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:08.862327    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc" event={"ID":"0e5c5023-9ed0-4471-865d-4f52eb8129ec","Type":"ContainerStarted","Data":"da98c21579af4bc0d9eabc1bbedd703b1bb0eb26b25fd11314f2bfb025a885a6"}
Sep 08 16:30:09.868957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:09.868919    2575 generic.go:356] "Generic (PLEG): container finished" podID="0e5c5023-9ed0-4471-865d-4f52eb8129ec" containerID="5a315bc761134a3943e44bc01b5cae8cfa9c6a2757a0a0e9b01a91d494c14529" exitCode=0
Sep 08 16:30:09.868957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:09.868959    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc" event={"ID":"0e5c5023-9ed0-4471-865d-4f52eb8129ec","Type":"ContainerDied","Data":"5a315bc761134a3943e44bc01b5cae8cfa9c6a2757a0a0e9b01a91d494c14529"}
Sep 08 16:30:11.011511 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:11.011485    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"
Sep 08 16:30:11.198260 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:11.198160    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5c5023-9ed0-4471-865d-4f52eb8129ec-maas-api-service-ca\") pod \"0e5c5023-9ed0-4471-865d-4f52eb8129ec\" (UID: \"0e5c5023-9ed0-4471-865d-4f52eb8129ec\") "
Sep 08 16:30:11.198526 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:11.198501    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5c5023-9ed0-4471-865d-4f52eb8129ec-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e5c5023-9ed0-4471-865d-4f52eb8129ec" (UID: "0e5c5023-9ed0-4471-865d-4f52eb8129ec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:30:11.299807 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:11.299770    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5c5023-9ed0-4471-865d-4f52eb8129ec-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:11.879838 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:11.879808    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"
Sep 08 16:30:11.879987 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:11.879844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc" event={"ID":"0e5c5023-9ed0-4471-865d-4f52eb8129ec","Type":"ContainerDied","Data":"da98c21579af4bc0d9eabc1bbedd703b1bb0eb26b25fd11314f2bfb025a885a6"}
Sep 08 16:30:11.879987 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:11.879890    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da98c21579af4bc0d9eabc1bbedd703b1bb0eb26b25fd11314f2bfb025a885a6"
Sep 08 16:30:12.815823 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:12.815790    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:30:14.837824 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:14.837792    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:30:19.509632 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.509562    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"]
Sep 08 16:30:19.512303 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.512271    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ec338109-29814750-qqpbw"]
Sep 08 16:30:19.547665 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.547634    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"]
Sep 08 16:30:19.547969 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.547879    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59" podUID="8b9edc65-e3dd-48d6-95d2-3bff0a739fb2" containerName="payload-processing" containerID="cri-o://00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a" gracePeriod=30
Sep 08 16:30:19.552931 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.552909    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"]
Sep 08 16:30:19.553412 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.553363    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r" podUID="749c1bdc-868c-4b0a-ac44-2c245389f83d" containerName="payload-pre-processing" containerID="cri-o://5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba" gracePeriod=30
Sep 08 16:30:19.792943 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.792863    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"]
Sep 08 16:30:19.797002 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.796975    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ec338109-2f7lc"]
Sep 08 16:30:19.825945 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.825848    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:30:19.835372 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.835351    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:30:19.870899 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.870877    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c1bdc-868c-4b0a-ac44-2c245389f83d-plugins-config-volume\") pod \"749c1bdc-868c-4b0a-ac44-2c245389f83d\" (UID: \"749c1bdc-868c-4b0a-ac44-2c245389f83d\") "
Sep 08 16:30:19.871022 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.870911    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95lk4\" (UniqueName: \"kubernetes.io/projected/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-kube-api-access-95lk4\") pod \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\" (UID: \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\") "
Sep 08 16:30:19.871022 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.870945    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-plugins-config-volume\") pod \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\" (UID: \"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2\") "
Sep 08 16:30:19.871137 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.871044    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh6nq\" (UniqueName: \"kubernetes.io/projected/749c1bdc-868c-4b0a-ac44-2c245389f83d-kube-api-access-sh6nq\") pod \"749c1bdc-868c-4b0a-ac44-2c245389f83d\" (UID: \"749c1bdc-868c-4b0a-ac44-2c245389f83d\") "
Sep 08 16:30:19.871302 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.871273    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749c1bdc-868c-4b0a-ac44-2c245389f83d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "749c1bdc-868c-4b0a-ac44-2c245389f83d" (UID: "749c1bdc-868c-4b0a-ac44-2c245389f83d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:30:19.871413 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.871330    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b9edc65-e3dd-48d6-95d2-3bff0a739fb2" (UID: "8b9edc65-e3dd-48d6-95d2-3bff0a739fb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:30:19.871413 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.871348    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749c1bdc-868c-4b0a-ac44-2c245389f83d-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:19.873009 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.872984    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-kube-api-access-95lk4" (OuterVolumeSpecName: "kube-api-access-95lk4") pod "8b9edc65-e3dd-48d6-95d2-3bff0a739fb2" (UID: "8b9edc65-e3dd-48d6-95d2-3bff0a739fb2"). InnerVolumeSpecName "kube-api-access-95lk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:30:19.873116 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.873091    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749c1bdc-868c-4b0a-ac44-2c245389f83d-kube-api-access-sh6nq" (OuterVolumeSpecName: "kube-api-access-sh6nq") pod "749c1bdc-868c-4b0a-ac44-2c245389f83d" (UID: "749c1bdc-868c-4b0a-ac44-2c245389f83d"). InnerVolumeSpecName "kube-api-access-sh6nq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:30:19.926690 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.926652    2575 generic.go:356] "Generic (PLEG): container finished" podID="8b9edc65-e3dd-48d6-95d2-3bff0a739fb2" containerID="00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a" exitCode=0
Sep 08 16:30:19.926847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.926695    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"
Sep 08 16:30:19.926847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.926698    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59" event={"ID":"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2","Type":"ContainerDied","Data":"00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a"}
Sep 08 16:30:19.926847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.926823    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59" event={"ID":"8b9edc65-e3dd-48d6-95d2-3bff0a739fb2","Type":"ContainerDied","Data":"5a446a61ceb7d55f4c09de430c7964511c227d6ef8de5cc70a65167e00dd880c"}
Sep 08 16:30:19.926847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.926840    2575 scope.go:117] "RemoveContainer" containerID="00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a"
Sep 08 16:30:19.928336 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.928311    2575 generic.go:356] "Generic (PLEG): container finished" podID="749c1bdc-868c-4b0a-ac44-2c245389f83d" containerID="5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba" exitCode=0
Sep 08 16:30:19.928450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.928341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r" event={"ID":"749c1bdc-868c-4b0a-ac44-2c245389f83d","Type":"ContainerDied","Data":"5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba"}
Sep 08 16:30:19.928450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.928384    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r" event={"ID":"749c1bdc-868c-4b0a-ac44-2c245389f83d","Type":"ContainerDied","Data":"6f435898183543a479641f001b681e454eb6e6efaf880cdf0a2a7359d266b1df"}
Sep 08 16:30:19.928631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.928608    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"
Sep 08 16:30:19.936730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.936710    2575 scope.go:117] "RemoveContainer" containerID="00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a"
Sep 08 16:30:19.937060 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:30:19.937028    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a\": container with ID starting with 00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a not found: ID does not exist" containerID="00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a"
Sep 08 16:30:19.937149 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.937070    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a"} err="failed to get container status \"00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a\": rpc error: code = NotFound desc = could not find container \"00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a\": container with ID starting with 00c888edb8889edf40d72fb3941c9b14be5d1517376781a2e8c65b56a320ad1a not found: ID does not exist"
Sep 08 16:30:19.937149 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.937095    2575 scope.go:117] "RemoveContainer" containerID="5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba"
Sep 08 16:30:19.948111 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.948095    2575 scope.go:117] "RemoveContainer" containerID="5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba"
Sep 08 16:30:19.948374 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:30:19.948359    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba\": container with ID starting with 5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba not found: ID does not exist" containerID="5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba"
Sep 08 16:30:19.948429 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.948379    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba"} err="failed to get container status \"5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba\": rpc error: code = NotFound desc = could not find container \"5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba\": container with ID starting with 5b5fcfd4190121459af73d4678c552e8ae09ede3f6cc7b07e17be66a6894e2ba not found: ID does not exist"
Sep 08 16:30:19.971870 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.971847    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6nq\" (UniqueName: \"kubernetes.io/projected/749c1bdc-868c-4b0a-ac44-2c245389f83d-kube-api-access-sh6nq\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:19.971870 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.971866    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95lk4\" (UniqueName: \"kubernetes.io/projected/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-kube-api-access-95lk4\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:19.971989 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.971875    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:19.986922 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.986899    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"]
Sep 08 16:30:19.996973 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:19.996954    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ec338109-84688d9b9f-7gw59"]
Sep 08 16:30:20.019856 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:20.019833    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"]
Sep 08 16:30:20.080754 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:20.080733    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ec338109-647fb6d99d-knd8r"]
Sep 08 16:30:20.507604 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:20.507518    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e5c5023-9ed0-4471-865d-4f52eb8129ec" path="/var/lib/kubelet/pods/0e5c5023-9ed0-4471-865d-4f52eb8129ec/volumes"
Sep 08 16:30:20.507824 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:20.507810    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="164442cb-e9fe-493a-a38e-1b6b98c74974" path="/var/lib/kubelet/pods/164442cb-e9fe-493a-a38e-1b6b98c74974/volumes"
Sep 08 16:30:20.508120 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:20.508104    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="749c1bdc-868c-4b0a-ac44-2c245389f83d" path="/var/lib/kubelet/pods/749c1bdc-868c-4b0a-ac44-2c245389f83d/volumes"
Sep 08 16:30:20.508421 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:20.508410    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b9edc65-e3dd-48d6-95d2-3bff0a739fb2" path="/var/lib/kubelet/pods/8b9edc65-e3dd-48d6-95d2-3bff0a739fb2/volumes"
Sep 08 16:30:24.472359 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472325    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"]
Sep 08 16:30:24.472763 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472749    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b9edc65-e3dd-48d6-95d2-3bff0a739fb2" containerName="payload-processing"
Sep 08 16:30:24.472808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472765    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b9edc65-e3dd-48d6-95d2-3bff0a739fb2" containerName="payload-processing"
Sep 08 16:30:24.472808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472772    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5c5023-9ed0-4471-865d-4f52eb8129ec" containerName="revoke-keys"
Sep 08 16:30:24.472808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472779    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5c5023-9ed0-4471-865d-4f52eb8129ec" containerName="revoke-keys"
Sep 08 16:30:24.472808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472792    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="749c1bdc-868c-4b0a-ac44-2c245389f83d" containerName="payload-pre-processing"
Sep 08 16:30:24.472808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472798    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="749c1bdc-868c-4b0a-ac44-2c245389f83d" containerName="payload-pre-processing"
Sep 08 16:30:24.472955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472855    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="749c1bdc-868c-4b0a-ac44-2c245389f83d" containerName="payload-pre-processing"
Sep 08 16:30:24.472955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472862    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b9edc65-e3dd-48d6-95d2-3bff0a739fb2" containerName="payload-processing"
Sep 08 16:30:24.472955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.472871    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e5c5023-9ed0-4471-865d-4f52eb8129ec" containerName="revoke-keys"
Sep 08 16:30:24.477566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.477537    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:24.480863 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.480838    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-8f31ec88\""
Sep 08 16:30:24.480974 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.480887    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-8f31ec88-dockercfg-k2lts\""
Sep 08 16:30:24.492772 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.492749    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"]
Sep 08 16:30:24.512682 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.512655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn\" (UID: \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\") " pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:24.512798 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.512776    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2579r\" (UniqueName: \"kubernetes.io/projected/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-kube-api-access-2579r\") pod \"payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn\" (UID: \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\") " pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:24.534067 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.534038    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"]
Sep 08 16:30:24.537345 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.537329    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:24.546717 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.546691    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"]
Sep 08 16:30:24.614274 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.614241    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2281e7ee-9922-4305-b30c-de50b42d12e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk\" (UID: \"2281e7ee-9922-4305-b30c-de50b42d12e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:24.614422 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.614294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2579r\" (UniqueName: \"kubernetes.io/projected/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-kube-api-access-2579r\") pod \"payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn\" (UID: \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\") " pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:24.614422 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.614393    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkls7\" (UniqueName: \"kubernetes.io/projected/2281e7ee-9922-4305-b30c-de50b42d12e8-kube-api-access-bkls7\") pod \"payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk\" (UID: \"2281e7ee-9922-4305-b30c-de50b42d12e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:24.614499 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.614447    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn\" (UID: \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\") " pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:24.615125 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.615107    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-plugins-config-volume\") pod \"payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn\" (UID: \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\") " pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:24.622884 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.622855    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2579r\" (UniqueName: \"kubernetes.io/projected/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-kube-api-access-2579r\") pod \"payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn\" (UID: \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\") " pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:24.714986 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.714954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2281e7ee-9922-4305-b30c-de50b42d12e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk\" (UID: \"2281e7ee-9922-4305-b30c-de50b42d12e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:24.715115 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.715075    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkls7\" (UniqueName: \"kubernetes.io/projected/2281e7ee-9922-4305-b30c-de50b42d12e8-kube-api-access-bkls7\") pod \"payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk\" (UID: \"2281e7ee-9922-4305-b30c-de50b42d12e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:24.715678 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.715656    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2281e7ee-9922-4305-b30c-de50b42d12e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk\" (UID: \"2281e7ee-9922-4305-b30c-de50b42d12e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:24.724752 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.724683    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkls7\" (UniqueName: \"kubernetes.io/projected/2281e7ee-9922-4305-b30c-de50b42d12e8-kube-api-access-bkls7\") pod \"payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk\" (UID: \"2281e7ee-9922-4305-b30c-de50b42d12e8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:24.789261 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.789232    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:24.848482 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:24.848116    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:25.071989 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.071963    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"]
Sep 08 16:30:25.073699 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:30:25.073675    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6baaa2_6e7b_43e0_8687_ae401abfbee9.slice/crio-3540553a896179fb400cc21caafc710a069e426cf75a33ae382ad7563f4d2c6a WatchSource:0}: Error finding container 3540553a896179fb400cc21caafc710a069e426cf75a33ae382ad7563f4d2c6a: Status 404 returned error can't find the container with id 3540553a896179fb400cc21caafc710a069e426cf75a33ae382ad7563f4d2c6a
Sep 08 16:30:25.141068 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.141040    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"]
Sep 08 16:30:25.141464 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:30:25.141441    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2281e7ee_9922_4305_b30c_de50b42d12e8.slice/crio-bf45293a890876cf1ebf5f696fd766d8a808cf21594183721bdee06bacc76cd4 WatchSource:0}: Error finding container bf45293a890876cf1ebf5f696fd766d8a808cf21594183721bdee06bacc76cd4: Status 404 returned error can't find the container with id bf45293a890876cf1ebf5f696fd766d8a808cf21594183721bdee06bacc76cd4
Sep 08 16:30:25.965262 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.965222    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk" event={"ID":"2281e7ee-9922-4305-b30c-de50b42d12e8","Type":"ContainerStarted","Data":"c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84"}
Sep 08 16:30:25.965632 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.965330    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk" event={"ID":"2281e7ee-9922-4305-b30c-de50b42d12e8","Type":"ContainerStarted","Data":"bf45293a890876cf1ebf5f696fd766d8a808cf21594183721bdee06bacc76cd4"}
Sep 08 16:30:25.965723 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.965699    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:25.967318 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.967292    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn" event={"ID":"3c6baaa2-6e7b-43e0-8687-ae401abfbee9","Type":"ContainerStarted","Data":"5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029"}
Sep 08 16:30:25.967441 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.967329    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn" event={"ID":"3c6baaa2-6e7b-43e0-8687-ae401abfbee9","Type":"ContainerStarted","Data":"3540553a896179fb400cc21caafc710a069e426cf75a33ae382ad7563f4d2c6a"}
Sep 08 16:30:25.967441 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.967367    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:25.983432 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.983388    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk" podStartSLOduration=1.98337454 podStartE2EDuration="1.98337454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:25.981677434 +0000 UTC m=+1060.068341183" watchObservedRunningTime="2026-09-08 16:30:25.98337454 +0000 UTC m=+1060.070038287"
Sep 08 16:30:25.999522 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:25.999479    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn" podStartSLOduration=1.9994668469999999 podStartE2EDuration="1.999466847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:25.996556242 +0000 UTC m=+1060.083219991" watchObservedRunningTime="2026-09-08 16:30:25.999466847 +0000 UTC m=+1060.086130595"
Sep 08 16:30:36.974052 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:36.974020    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:36.974521 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:36.974100    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:48.624704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:48.624675    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"]
Sep 08 16:30:48.626990 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:48.626973    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"
Sep 08 16:30:48.629740 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:48.629719    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:30:48.635433 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:48.635402    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"]
Sep 08 16:30:48.722076 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:48.722038    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da69dad-cf13-4784-9e9b-78d66e358131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx\" (UID: \"8da69dad-cf13-4784-9e9b-78d66e358131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"
Sep 08 16:30:48.822568 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:48.822535    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da69dad-cf13-4784-9e9b-78d66e358131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx\" (UID: \"8da69dad-cf13-4784-9e9b-78d66e358131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"
Sep 08 16:30:48.823343 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:48.823318    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da69dad-cf13-4784-9e9b-78d66e358131-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx\" (UID: \"8da69dad-cf13-4784-9e9b-78d66e358131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"
Sep 08 16:30:48.938315 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:48.938251    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"
Sep 08 16:30:49.215077 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:49.215046    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"]
Sep 08 16:30:49.215850 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:30:49.215818    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da69dad_cf13_4784_9e9b_78d66e358131.slice/crio-6f08658efef7d4b6467d2db5dcf942ba6e3466770822ec9897d41284367678bd WatchSource:0}: Error finding container 6f08658efef7d4b6467d2db5dcf942ba6e3466770822ec9897d41284367678bd: Status 404 returned error can't find the container with id 6f08658efef7d4b6467d2db5dcf942ba6e3466770822ec9897d41284367678bd
Sep 08 16:30:50.094591 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:50.094552    2575 generic.go:356] "Generic (PLEG): container finished" podID="8da69dad-cf13-4784-9e9b-78d66e358131" containerID="e9215d9f09e128e509e4290f40d9a54fe4100b5ee02837fbb856198e273673c2" exitCode=0
Sep 08 16:30:50.094946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:50.094646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx" event={"ID":"8da69dad-cf13-4784-9e9b-78d66e358131","Type":"ContainerDied","Data":"e9215d9f09e128e509e4290f40d9a54fe4100b5ee02837fbb856198e273673c2"}
Sep 08 16:30:50.094946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:50.094719    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx" event={"ID":"8da69dad-cf13-4784-9e9b-78d66e358131","Type":"ContainerStarted","Data":"6f08658efef7d4b6467d2db5dcf942ba6e3466770822ec9897d41284367678bd"}
Sep 08 16:30:51.235907 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:51.235884    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"
Sep 08 16:30:51.339648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:51.339619    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da69dad-cf13-4784-9e9b-78d66e358131-maas-api-service-ca\") pod \"8da69dad-cf13-4784-9e9b-78d66e358131\" (UID: \"8da69dad-cf13-4784-9e9b-78d66e358131\") "
Sep 08 16:30:51.339939 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:51.339917    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da69dad-cf13-4784-9e9b-78d66e358131-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8da69dad-cf13-4784-9e9b-78d66e358131" (UID: "8da69dad-cf13-4784-9e9b-78d66e358131"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:30:51.440681 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:51.440604    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da69dad-cf13-4784-9e9b-78d66e358131-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:52.107733 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:52.107700    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx" event={"ID":"8da69dad-cf13-4784-9e9b-78d66e358131","Type":"ContainerDied","Data":"6f08658efef7d4b6467d2db5dcf942ba6e3466770822ec9897d41284367678bd"}
Sep 08 16:30:52.107733 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:52.107725    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"
Sep 08 16:30:52.107930 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:52.107751    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f08658efef7d4b6467d2db5dcf942ba6e3466770822ec9897d41284367678bd"
Sep 08 16:30:54.673269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:54.673236    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"]
Sep 08 16:30:54.675161 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:54.675092    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn" podUID="3c6baaa2-6e7b-43e0-8687-ae401abfbee9" containerName="payload-processing" containerID="cri-o://5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029" gracePeriod=30
Sep 08 16:30:54.678538 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:54.678510    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"]
Sep 08 16:30:54.678870 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:54.678843    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk" podUID="2281e7ee-9922-4305-b30c-de50b42d12e8" containerName="payload-pre-processing" containerID="cri-o://c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84" gracePeriod=30
Sep 08 16:30:54.740728 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:30:54.740686    2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6baaa2_6e7b_43e0_8687_ae401abfbee9.slice/crio-5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2281e7ee_9922_4305_b30c_de50b42d12e8.slice/crio-c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2281e7ee_9922_4305_b30c_de50b42d12e8.slice/crio-conmon-c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84.scope\": RecentStats: unable to find data in memory cache]"
Sep 08 16:30:54.824071 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:54.824022    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"]
Sep 08 16:30:54.826655 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:54.826624    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-8f31ec88-tkcsx"]
Sep 08 16:30:54.951999 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:54.951974    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:54.979996 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:54.979964    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:55.075003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.074964    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2579r\" (UniqueName: \"kubernetes.io/projected/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-kube-api-access-2579r\") pod \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\" (UID: \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\") "
Sep 08 16:30:55.075003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.075006    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2281e7ee-9922-4305-b30c-de50b42d12e8-plugins-config-volume\") pod \"2281e7ee-9922-4305-b30c-de50b42d12e8\" (UID: \"2281e7ee-9922-4305-b30c-de50b42d12e8\") "
Sep 08 16:30:55.075255 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.075041    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-plugins-config-volume\") pod \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\" (UID: \"3c6baaa2-6e7b-43e0-8687-ae401abfbee9\") "
Sep 08 16:30:55.075255 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.075104    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkls7\" (UniqueName: \"kubernetes.io/projected/2281e7ee-9922-4305-b30c-de50b42d12e8-kube-api-access-bkls7\") pod \"2281e7ee-9922-4305-b30c-de50b42d12e8\" (UID: \"2281e7ee-9922-4305-b30c-de50b42d12e8\") "
Sep 08 16:30:55.075396 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.075357    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2281e7ee-9922-4305-b30c-de50b42d12e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2281e7ee-9922-4305-b30c-de50b42d12e8" (UID: "2281e7ee-9922-4305-b30c-de50b42d12e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:30:55.075460 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.075428    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c6baaa2-6e7b-43e0-8687-ae401abfbee9" (UID: "3c6baaa2-6e7b-43e0-8687-ae401abfbee9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:30:55.077186 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.077163    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-kube-api-access-2579r" (OuterVolumeSpecName: "kube-api-access-2579r") pod "3c6baaa2-6e7b-43e0-8687-ae401abfbee9" (UID: "3c6baaa2-6e7b-43e0-8687-ae401abfbee9"). InnerVolumeSpecName "kube-api-access-2579r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:30:55.077287 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.077192    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2281e7ee-9922-4305-b30c-de50b42d12e8-kube-api-access-bkls7" (OuterVolumeSpecName: "kube-api-access-bkls7") pod "2281e7ee-9922-4305-b30c-de50b42d12e8" (UID: "2281e7ee-9922-4305-b30c-de50b42d12e8"). InnerVolumeSpecName "kube-api-access-bkls7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:30:55.128091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.128049    2575 generic.go:356] "Generic (PLEG): container finished" podID="2281e7ee-9922-4305-b30c-de50b42d12e8" containerID="c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84" exitCode=0
Sep 08 16:30:55.128253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.128105    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"
Sep 08 16:30:55.128253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.128139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk" event={"ID":"2281e7ee-9922-4305-b30c-de50b42d12e8","Type":"ContainerDied","Data":"c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84"}
Sep 08 16:30:55.128253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.128203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk" event={"ID":"2281e7ee-9922-4305-b30c-de50b42d12e8","Type":"ContainerDied","Data":"bf45293a890876cf1ebf5f696fd766d8a808cf21594183721bdee06bacc76cd4"}
Sep 08 16:30:55.128253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.128233    2575 scope.go:117] "RemoveContainer" containerID="c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84"
Sep 08 16:30:55.130076 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.130052    2575 generic.go:356] "Generic (PLEG): container finished" podID="3c6baaa2-6e7b-43e0-8687-ae401abfbee9" containerID="5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029" exitCode=0
Sep 08 16:30:55.130199 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.130091    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn" event={"ID":"3c6baaa2-6e7b-43e0-8687-ae401abfbee9","Type":"ContainerDied","Data":"5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029"}
Sep 08 16:30:55.130199 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.130114    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"
Sep 08 16:30:55.130199 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.130139    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn" event={"ID":"3c6baaa2-6e7b-43e0-8687-ae401abfbee9","Type":"ContainerDied","Data":"3540553a896179fb400cc21caafc710a069e426cf75a33ae382ad7563f4d2c6a"}
Sep 08 16:30:55.142235 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.142214    2575 scope.go:117] "RemoveContainer" containerID="c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84"
Sep 08 16:30:55.142516 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:30:55.142496    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84\": container with ID starting with c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84 not found: ID does not exist" containerID="c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84"
Sep 08 16:30:55.142730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.142529    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84"} err="failed to get container status \"c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84\": rpc error: code = NotFound desc = could not find container \"c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84\": container with ID starting with c9626f8f771570deba293cf4f7edf0dc6a3e37f021792106a1bf199e229b0b84 not found: ID does not exist"
Sep 08 16:30:55.142836 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.142736    2575 scope.go:117] "RemoveContainer" containerID="5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029"
Sep 08 16:30:55.151995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.151973    2575 scope.go:117] "RemoveContainer" containerID="5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029"
Sep 08 16:30:55.152264 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:30:55.152245    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029\": container with ID starting with 5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029 not found: ID does not exist" containerID="5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029"
Sep 08 16:30:55.152322 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.152274    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029"} err="failed to get container status \"5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029\": rpc error: code = NotFound desc = could not find container \"5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029\": container with ID starting with 5f2ea734540b407f8496e27447aee705b993f04b4205d833bff082bdaa875029 not found: ID does not exist"
Sep 08 16:30:55.155481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.155454    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"]
Sep 08 16:30:55.165807 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.165782    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-8f31ec88-76fcdfcb78-xbmnk"]
Sep 08 16:30:55.176157 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.176126    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2579r\" (UniqueName: \"kubernetes.io/projected/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-kube-api-access-2579r\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:55.176157 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.176159    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2281e7ee-9922-4305-b30c-de50b42d12e8-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:55.176297 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.176173    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c6baaa2-6e7b-43e0-8687-ae401abfbee9-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:55.176297 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.176185    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkls7\" (UniqueName: \"kubernetes.io/projected/2281e7ee-9922-4305-b30c-de50b42d12e8-kube-api-access-bkls7\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:30:55.182391 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.182329    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"]
Sep 08 16:30:55.185950 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:55.185929    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-8f31ec88-7b46b546fd-f6qrn"]
Sep 08 16:30:56.507747 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:56.507707    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2281e7ee-9922-4305-b30c-de50b42d12e8" path="/var/lib/kubelet/pods/2281e7ee-9922-4305-b30c-de50b42d12e8/volumes"
Sep 08 16:30:56.508247 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:56.508227    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c6baaa2-6e7b-43e0-8687-ae401abfbee9" path="/var/lib/kubelet/pods/3c6baaa2-6e7b-43e0-8687-ae401abfbee9/volumes"
Sep 08 16:30:56.508599 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:56.508567    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da69dad-cf13-4784-9e9b-78d66e358131" path="/var/lib/kubelet/pods/8da69dad-cf13-4784-9e9b-78d66e358131/volumes"
Sep 08 16:30:57.464508 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.464474    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"]
Sep 08 16:30:57.464893 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.464881    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6baaa2-6e7b-43e0-8687-ae401abfbee9" containerName="payload-processing"
Sep 08 16:30:57.464942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.464894    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6baaa2-6e7b-43e0-8687-ae401abfbee9" containerName="payload-processing"
Sep 08 16:30:57.464942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.464916    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2281e7ee-9922-4305-b30c-de50b42d12e8" containerName="payload-pre-processing"
Sep 08 16:30:57.464942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.464922    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2281e7ee-9922-4305-b30c-de50b42d12e8" containerName="payload-pre-processing"
Sep 08 16:30:57.464942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.464935    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da69dad-cf13-4784-9e9b-78d66e358131" containerName="revoke-keys"
Sep 08 16:30:57.464942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.464941    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da69dad-cf13-4784-9e9b-78d66e358131" containerName="revoke-keys"
Sep 08 16:30:57.465122 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.465004    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c6baaa2-6e7b-43e0-8687-ae401abfbee9" containerName="payload-processing"
Sep 08 16:30:57.465122 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.465012    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da69dad-cf13-4784-9e9b-78d66e358131" containerName="revoke-keys"
Sep 08 16:30:57.465122 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.465019    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2281e7ee-9922-4305-b30c-de50b42d12e8" containerName="payload-pre-processing"
Sep 08 16:30:57.467725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.467705    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.470276 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.470258    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-ec6a8395-kserve-self-signed-certs\""
Sep 08 16:30:57.476893 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.476867    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"]
Sep 08 16:30:57.601089 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.601052    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-tmp-dir\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.601089 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.601089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-model-cache\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.601636 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.601218    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-dshm\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.601636 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.601262    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b20085-9d99-4c68-91fc-24e589dfef81-tls-certs\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.601636 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.601327    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-home\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.601636 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.601442    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpmd\" (UniqueName: \"kubernetes.io/projected/76b20085-9d99-4c68-91fc-24e589dfef81-kube-api-access-wcpmd\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.702858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.702808    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpmd\" (UniqueName: \"kubernetes.io/projected/76b20085-9d99-4c68-91fc-24e589dfef81-kube-api-access-wcpmd\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.703050 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.702912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-tmp-dir\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.703050 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.702935    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-model-cache\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.703050 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.702957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-dshm\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.703050 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.702972    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b20085-9d99-4c68-91fc-24e589dfef81-tls-certs\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.703050 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.703001    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-home\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.703324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.703305    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-model-cache\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.703378 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.703335    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-home\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.703469 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.703446    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-tmp-dir\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.705405 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.705382    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-dshm\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.705608 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.705592    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b20085-9d99-4c68-91fc-24e589dfef81-tls-certs\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.710785 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.710762    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpmd\" (UniqueName: \"kubernetes.io/projected/76b20085-9d99-4c68-91fc-24e589dfef81-kube-api-access-wcpmd\") pod \"e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") " pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:57.777250 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:57.777165    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:30:58.070144 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:58.070103    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"]
Sep 08 16:30:58.152193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:58.152158    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv" event={"ID":"76b20085-9d99-4c68-91fc-24e589dfef81","Type":"ContainerStarted","Data":"de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721"}
Sep 08 16:30:58.152545 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:58.152203    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv" event={"ID":"76b20085-9d99-4c68-91fc-24e589dfef81","Type":"ContainerStarted","Data":"92b1214e3ddb08af2a812bb3018d4b83225fb803951c897a002232eef1237ff0"}
Sep 08 16:30:58.173518 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:30:58.173469    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv" podStartSLOduration=1.173452768 podStartE2EDuration="1.173452768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:30:58.170941835 +0000 UTC m=+1092.257605597" watchObservedRunningTime="2026-09-08 16:30:58.173452768 +0000 UTC m=+1092.260116549"
Sep 08 16:31:00.004808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.004770    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"]
Sep 08 16:31:00.007394 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.007370    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:00.008853 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.008833    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"]
Sep 08 16:31:00.011260 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.011244    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:00.011764 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.011748    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-aeb37378\""
Sep 08 16:31:00.023786 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.023762    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"]
Sep 08 16:31:00.033168 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.033139    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"]
Sep 08 16:31:00.127053 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.127006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkm7f\" (UniqueName: \"kubernetes.io/projected/ea40bba9-e35b-42b6-b3ef-5986348e4f84-kube-api-access-mkm7f\") pod \"payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9\" (UID: \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:00.127233 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.127132    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh876\" (UniqueName: \"kubernetes.io/projected/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-kube-api-access-sh876\") pod \"payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6\" (UID: \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\") " pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:00.127597 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.127546    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-plugins-config-volume\") pod \"payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6\" (UID: \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\") " pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:00.127748 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.127726    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40bba9-e35b-42b6-b3ef-5986348e4f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9\" (UID: \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:00.229234 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.229192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh876\" (UniqueName: \"kubernetes.io/projected/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-kube-api-access-sh876\") pod \"payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6\" (UID: \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\") " pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:00.229432 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.229241    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-plugins-config-volume\") pod \"payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6\" (UID: \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\") " pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:00.229432 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.229316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40bba9-e35b-42b6-b3ef-5986348e4f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9\" (UID: \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:00.229432 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.229375    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkm7f\" (UniqueName: \"kubernetes.io/projected/ea40bba9-e35b-42b6-b3ef-5986348e4f84-kube-api-access-mkm7f\") pod \"payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9\" (UID: \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:00.229999 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.229976    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-plugins-config-volume\") pod \"payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6\" (UID: \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\") " pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:00.230062 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.230032    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40bba9-e35b-42b6-b3ef-5986348e4f84-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9\" (UID: \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:00.241089 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.241061    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkm7f\" (UniqueName: \"kubernetes.io/projected/ea40bba9-e35b-42b6-b3ef-5986348e4f84-kube-api-access-mkm7f\") pod \"payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9\" (UID: \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:00.242450 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.242424    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh876\" (UniqueName: \"kubernetes.io/projected/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-kube-api-access-sh876\") pod \"payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6\" (UID: \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\") " pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:00.319737 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.319633    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:00.327168 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.327135    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:00.620127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.620087    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"]
Sep 08 16:31:00.630730 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:31:00.630698    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea40bba9_e35b_42b6_b3ef_5986348e4f84.slice/crio-bcb121b9acb4bd484269977787532e7ab8fdbdbd42be4dd27b714a8db7d8cd73 WatchSource:0}: Error finding container bcb121b9acb4bd484269977787532e7ab8fdbdbd42be4dd27b714a8db7d8cd73: Status 404 returned error can't find the container with id bcb121b9acb4bd484269977787532e7ab8fdbdbd42be4dd27b714a8db7d8cd73
Sep 08 16:31:00.634830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:00.634807    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"]
Sep 08 16:31:00.637263 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:31:00.637238    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb4b8a7_468c_4d28_989b_2c52ca3c598c.slice/crio-818917c93efea8f38808c8871eb6f18a016fa50d4756f7fef07d720c28883446 WatchSource:0}: Error finding container 818917c93efea8f38808c8871eb6f18a016fa50d4756f7fef07d720c28883446: Status 404 returned error can't find the container with id 818917c93efea8f38808c8871eb6f18a016fa50d4756f7fef07d720c28883446
Sep 08 16:31:01.174014 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:01.173973    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9" event={"ID":"ea40bba9-e35b-42b6-b3ef-5986348e4f84","Type":"ContainerStarted","Data":"e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405"}
Sep 08 16:31:01.174472 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:01.174049    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:01.174472 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:01.174078    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9" event={"ID":"ea40bba9-e35b-42b6-b3ef-5986348e4f84","Type":"ContainerStarted","Data":"bcb121b9acb4bd484269977787532e7ab8fdbdbd42be4dd27b714a8db7d8cd73"}
Sep 08 16:31:01.175702 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:01.175671    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6" event={"ID":"6cb4b8a7-468c-4d28-989b-2c52ca3c598c","Type":"ContainerStarted","Data":"574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220"}
Sep 08 16:31:01.175829 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:01.175725    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6" event={"ID":"6cb4b8a7-468c-4d28-989b-2c52ca3c598c","Type":"ContainerStarted","Data":"818917c93efea8f38808c8871eb6f18a016fa50d4756f7fef07d720c28883446"}
Sep 08 16:31:01.175829 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:01.175762    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:01.196614 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:01.196548    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9" podStartSLOduration=2.196532415 podStartE2EDuration="2.196532415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:01.194465706 +0000 UTC m=+1095.281129456" watchObservedRunningTime="2026-09-08 16:31:01.196532415 +0000 UTC m=+1095.283196162"
Sep 08 16:31:01.217840 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:01.217780    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6" podStartSLOduration=2.217753879 podStartE2EDuration="2.217753879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:01.214370537 +0000 UTC m=+1095.301034286" watchObservedRunningTime="2026-09-08 16:31:01.217753879 +0000 UTC m=+1095.304417627"
Sep 08 16:31:07.777796 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:07.777765    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:31:07.778238 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:07.778005    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:31:07.790285 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:07.790260    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:31:08.224054 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:08.224023    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:31:12.184158 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:12.184120    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:12.184630 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:12.184200    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:23.567847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:23.567813    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"]
Sep 08 16:31:23.570698 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:23.570678    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"
Sep 08 16:31:23.573623 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:23.573601    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:31:23.581329 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:23.581305    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"]
Sep 08 16:31:23.643461 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:23.643429    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10efce-d920-4c77-b6b2-53d56f5581bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aeb37378-wth9c\" (UID: \"ae10efce-d920-4c77-b6b2-53d56f5581bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"
Sep 08 16:31:23.744191 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:23.744161    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10efce-d920-4c77-b6b2-53d56f5581bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aeb37378-wth9c\" (UID: \"ae10efce-d920-4c77-b6b2-53d56f5581bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"
Sep 08 16:31:23.744764 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:23.744745    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10efce-d920-4c77-b6b2-53d56f5581bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-aeb37378-wth9c\" (UID: \"ae10efce-d920-4c77-b6b2-53d56f5581bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"
Sep 08 16:31:23.883315 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:23.883283    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"
Sep 08 16:31:24.156982 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:24.156956    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"]
Sep 08 16:31:24.158840 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:31:24.158809    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae10efce_d920_4c77_b6b2_53d56f5581bb.slice/crio-cc2f9cc1a7ec5a563f713fdf6193cf740a57203565357a9421eef562824c268d WatchSource:0}: Error finding container cc2f9cc1a7ec5a563f713fdf6193cf740a57203565357a9421eef562824c268d: Status 404 returned error can't find the container with id cc2f9cc1a7ec5a563f713fdf6193cf740a57203565357a9421eef562824c268d
Sep 08 16:31:24.305076 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:24.305048    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c" event={"ID":"ae10efce-d920-4c77-b6b2-53d56f5581bb","Type":"ContainerStarted","Data":"cc2f9cc1a7ec5a563f713fdf6193cf740a57203565357a9421eef562824c268d"}
Sep 08 16:31:25.312926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:25.312892    2575 generic.go:356] "Generic (PLEG): container finished" podID="ae10efce-d920-4c77-b6b2-53d56f5581bb" containerID="f89da7d54057e0b57fcd2bee1306c78835159d8b1ba97c0578412496ea713cff" exitCode=0
Sep 08 16:31:25.313261 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:25.312946    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c" event={"ID":"ae10efce-d920-4c77-b6b2-53d56f5581bb","Type":"ContainerDied","Data":"f89da7d54057e0b57fcd2bee1306c78835159d8b1ba97c0578412496ea713cff"}
Sep 08 16:31:26.457520 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:26.457496    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"
Sep 08 16:31:26.570568 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:26.570479    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10efce-d920-4c77-b6b2-53d56f5581bb-maas-api-service-ca\") pod \"ae10efce-d920-4c77-b6b2-53d56f5581bb\" (UID: \"ae10efce-d920-4c77-b6b2-53d56f5581bb\") "
Sep 08 16:31:26.570878 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:26.570854    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae10efce-d920-4c77-b6b2-53d56f5581bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae10efce-d920-4c77-b6b2-53d56f5581bb" (UID: "ae10efce-d920-4c77-b6b2-53d56f5581bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:31:26.672091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:26.672049    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae10efce-d920-4c77-b6b2-53d56f5581bb-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:31:27.327234 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:27.327206    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"
Sep 08 16:31:27.327402 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:27.327250    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c" event={"ID":"ae10efce-d920-4c77-b6b2-53d56f5581bb","Type":"ContainerDied","Data":"cc2f9cc1a7ec5a563f713fdf6193cf740a57203565357a9421eef562824c268d"}
Sep 08 16:31:27.327402 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:27.327299    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc2f9cc1a7ec5a563f713fdf6193cf740a57203565357a9421eef562824c268d"
Sep 08 16:31:29.874545 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:29.874507    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"]
Sep 08 16:31:29.875096 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:29.874795    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6" podUID="6cb4b8a7-468c-4d28-989b-2c52ca3c598c" containerName="payload-processing" containerID="cri-o://574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220" gracePeriod=30
Sep 08 16:31:29.911320 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:29.911291    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"]
Sep 08 16:31:29.911536 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:29.911492    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9" podUID="ea40bba9-e35b-42b6-b3ef-5986348e4f84" containerName="payload-pre-processing" containerID="cri-o://e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405" gracePeriod=30
Sep 08 16:31:30.181033 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.181001    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:30.184706 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.184685    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:30.306647 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.306608    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkm7f\" (UniqueName: \"kubernetes.io/projected/ea40bba9-e35b-42b6-b3ef-5986348e4f84-kube-api-access-mkm7f\") pod \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\" (UID: \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\") "
Sep 08 16:31:30.306835 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.306706    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh876\" (UniqueName: \"kubernetes.io/projected/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-kube-api-access-sh876\") pod \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\" (UID: \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\") "
Sep 08 16:31:30.306835 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.306747    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-plugins-config-volume\") pod \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\" (UID: \"6cb4b8a7-468c-4d28-989b-2c52ca3c598c\") "
Sep 08 16:31:30.306835 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.306779    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40bba9-e35b-42b6-b3ef-5986348e4f84-plugins-config-volume\") pod \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\" (UID: \"ea40bba9-e35b-42b6-b3ef-5986348e4f84\") "
Sep 08 16:31:30.307145 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.307122    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6cb4b8a7-468c-4d28-989b-2c52ca3c598c" (UID: "6cb4b8a7-468c-4d28-989b-2c52ca3c598c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:31:30.307218 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.307150    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea40bba9-e35b-42b6-b3ef-5986348e4f84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea40bba9-e35b-42b6-b3ef-5986348e4f84" (UID: "ea40bba9-e35b-42b6-b3ef-5986348e4f84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:31:30.308740 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.308717    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea40bba9-e35b-42b6-b3ef-5986348e4f84-kube-api-access-mkm7f" (OuterVolumeSpecName: "kube-api-access-mkm7f") pod "ea40bba9-e35b-42b6-b3ef-5986348e4f84" (UID: "ea40bba9-e35b-42b6-b3ef-5986348e4f84"). InnerVolumeSpecName "kube-api-access-mkm7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:31:30.308913 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.308889    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-kube-api-access-sh876" (OuterVolumeSpecName: "kube-api-access-sh876") pod "6cb4b8a7-468c-4d28-989b-2c52ca3c598c" (UID: "6cb4b8a7-468c-4d28-989b-2c52ca3c598c"). InnerVolumeSpecName "kube-api-access-sh876". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:31:30.345896 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.345871    2575 generic.go:356] "Generic (PLEG): container finished" podID="6cb4b8a7-468c-4d28-989b-2c52ca3c598c" containerID="574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220" exitCode=0
Sep 08 16:31:30.346010 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.345915    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"
Sep 08 16:31:30.346010 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.345963    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6" event={"ID":"6cb4b8a7-468c-4d28-989b-2c52ca3c598c","Type":"ContainerDied","Data":"574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220"}
Sep 08 16:31:30.346126 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.346021    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6" event={"ID":"6cb4b8a7-468c-4d28-989b-2c52ca3c598c","Type":"ContainerDied","Data":"818917c93efea8f38808c8871eb6f18a016fa50d4756f7fef07d720c28883446"}
Sep 08 16:31:30.346126 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.346053    2575 scope.go:117] "RemoveContainer" containerID="574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220"
Sep 08 16:31:30.347367 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.347347    2575 generic.go:356] "Generic (PLEG): container finished" podID="ea40bba9-e35b-42b6-b3ef-5986348e4f84" containerID="e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405" exitCode=0
Sep 08 16:31:30.347463 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.347388    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"
Sep 08 16:31:30.347463 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.347429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9" event={"ID":"ea40bba9-e35b-42b6-b3ef-5986348e4f84","Type":"ContainerDied","Data":"e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405"}
Sep 08 16:31:30.347571 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.347477    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9" event={"ID":"ea40bba9-e35b-42b6-b3ef-5986348e4f84","Type":"ContainerDied","Data":"bcb121b9acb4bd484269977787532e7ab8fdbdbd42be4dd27b714a8db7d8cd73"}
Sep 08 16:31:30.355600 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.355570    2575 scope.go:117] "RemoveContainer" containerID="574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220"
Sep 08 16:31:30.355930 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:31:30.355900    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220\": container with ID starting with 574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220 not found: ID does not exist" containerID="574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220"
Sep 08 16:31:30.356012 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.355938    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220"} err="failed to get container status \"574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220\": rpc error: code = NotFound desc = could not find container \"574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220\": container with ID starting with 574225e4cd1f7da42bce8f65cd7022055e01f61b8b04583964dee9b9ff64c220 not found: ID does not exist"
Sep 08 16:31:30.356012 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.355964    2575 scope.go:117] "RemoveContainer" containerID="e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405"
Sep 08 16:31:30.363451 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.363431    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"]
Sep 08 16:31:30.364543 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.364522    2575 scope.go:117] "RemoveContainer" containerID="e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405"
Sep 08 16:31:30.364936 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:31:30.364914    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405\": container with ID starting with e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405 not found: ID does not exist" containerID="e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405"
Sep 08 16:31:30.365019 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.364944    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405"} err="failed to get container status \"e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405\": rpc error: code = NotFound desc = could not find container \"e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405\": container with ID starting with e10296f2d61c12aa0c1362ba83744d3d0038229fabf0e9050cc63ddaceabd405 not found: ID does not exist"
Sep 08 16:31:30.367320 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.367299    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-aeb37378-wth9c"]
Sep 08 16:31:30.382791 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.382771    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"]
Sep 08 16:31:30.396811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.396789    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-aeb37378-5c55489f98-98xk9"]
Sep 08 16:31:30.409808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.409750    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh876\" (UniqueName: \"kubernetes.io/projected/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-kube-api-access-sh876\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:31:30.409808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.409776    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb4b8a7-468c-4d28-989b-2c52ca3c598c-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:31:30.409808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.409787    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40bba9-e35b-42b6-b3ef-5986348e4f84-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:31:30.409808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.409796    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkm7f\" (UniqueName: \"kubernetes.io/projected/ea40bba9-e35b-42b6-b3ef-5986348e4f84-kube-api-access-mkm7f\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:31:30.423297 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.423266    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"]
Sep 08 16:31:30.428856 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.428835    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-aeb37378-584f7bc987-6gjg6"]
Sep 08 16:31:30.509936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.509902    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cb4b8a7-468c-4d28-989b-2c52ca3c598c" path="/var/lib/kubelet/pods/6cb4b8a7-468c-4d28-989b-2c52ca3c598c/volumes"
Sep 08 16:31:30.510429 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.510402    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae10efce-d920-4c77-b6b2-53d56f5581bb" path="/var/lib/kubelet/pods/ae10efce-d920-4c77-b6b2-53d56f5581bb/volumes"
Sep 08 16:31:30.511372 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:30.510987    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea40bba9-e35b-42b6-b3ef-5986348e4f84" path="/var/lib/kubelet/pods/ea40bba9-e35b-42b6-b3ef-5986348e4f84/volumes"
Sep 08 16:31:36.724758 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.724718    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"]
Sep 08 16:31:36.725198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725109    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb4b8a7-468c-4d28-989b-2c52ca3c598c" containerName="payload-processing"
Sep 08 16:31:36.725198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725120    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb4b8a7-468c-4d28-989b-2c52ca3c598c" containerName="payload-processing"
Sep 08 16:31:36.725198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725129    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae10efce-d920-4c77-b6b2-53d56f5581bb" containerName="revoke-keys"
Sep 08 16:31:36.725198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725135    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae10efce-d920-4c77-b6b2-53d56f5581bb" containerName="revoke-keys"
Sep 08 16:31:36.725198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725150    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea40bba9-e35b-42b6-b3ef-5986348e4f84" containerName="payload-pre-processing"
Sep 08 16:31:36.725198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725156    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea40bba9-e35b-42b6-b3ef-5986348e4f84" containerName="payload-pre-processing"
Sep 08 16:31:36.725479 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725215    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb4b8a7-468c-4d28-989b-2c52ca3c598c" containerName="payload-processing"
Sep 08 16:31:36.725479 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725228    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae10efce-d920-4c77-b6b2-53d56f5581bb" containerName="revoke-keys"
Sep 08 16:31:36.725479 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.725233    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea40bba9-e35b-42b6-b3ef-5986348e4f84" containerName="payload-pre-processing"
Sep 08 16:31:36.728081 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.728053    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:36.730677 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.730653    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-1b78b587-dockercfg-dnxnr\""
Sep 08 16:31:36.730938 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.730919    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-1b78b587\""
Sep 08 16:31:36.742601 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.742566    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"]
Sep 08 16:31:36.792644 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.792609    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"]
Sep 08 16:31:36.795360 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.795338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:36.814380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.814360    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"]
Sep 08 16:31:36.870839 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.870812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v4d2\" (UniqueName: \"kubernetes.io/projected/3839f612-8700-4b14-ba74-357592033488-kube-api-access-9v4d2\") pod \"payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4\" (UID: \"3839f612-8700-4b14-ba74-357592033488\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:36.870936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.870848    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3839f612-8700-4b14-ba74-357592033488-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4\" (UID: \"3839f612-8700-4b14-ba74-357592033488\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:36.971773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.971744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862210c8-a55c-4d92-82d2-4cfd081467aa-plugins-config-volume\") pod \"payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h\" (UID: \"862210c8-a55c-4d92-82d2-4cfd081467aa\") " pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:36.971889 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.971801    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v4d2\" (UniqueName: \"kubernetes.io/projected/3839f612-8700-4b14-ba74-357592033488-kube-api-access-9v4d2\") pod \"payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4\" (UID: \"3839f612-8700-4b14-ba74-357592033488\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:36.971889 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.971865    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3839f612-8700-4b14-ba74-357592033488-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4\" (UID: \"3839f612-8700-4b14-ba74-357592033488\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:36.972002 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.971895    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vf5c\" (UniqueName: \"kubernetes.io/projected/862210c8-a55c-4d92-82d2-4cfd081467aa-kube-api-access-2vf5c\") pod \"payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h\" (UID: \"862210c8-a55c-4d92-82d2-4cfd081467aa\") " pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:36.972480 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.972459    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3839f612-8700-4b14-ba74-357592033488-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4\" (UID: \"3839f612-8700-4b14-ba74-357592033488\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:36.980985 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:36.980938    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v4d2\" (UniqueName: \"kubernetes.io/projected/3839f612-8700-4b14-ba74-357592033488-kube-api-access-9v4d2\") pod \"payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4\" (UID: \"3839f612-8700-4b14-ba74-357592033488\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:37.038854 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.038831    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:37.072654 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.072626    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862210c8-a55c-4d92-82d2-4cfd081467aa-plugins-config-volume\") pod \"payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h\" (UID: \"862210c8-a55c-4d92-82d2-4cfd081467aa\") " pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:37.072783 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.072681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vf5c\" (UniqueName: \"kubernetes.io/projected/862210c8-a55c-4d92-82d2-4cfd081467aa-kube-api-access-2vf5c\") pod \"payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h\" (UID: \"862210c8-a55c-4d92-82d2-4cfd081467aa\") " pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:37.073342 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.073312    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862210c8-a55c-4d92-82d2-4cfd081467aa-plugins-config-volume\") pod \"payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h\" (UID: \"862210c8-a55c-4d92-82d2-4cfd081467aa\") " pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:37.080724 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.080700    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vf5c\" (UniqueName: \"kubernetes.io/projected/862210c8-a55c-4d92-82d2-4cfd081467aa-kube-api-access-2vf5c\") pod \"payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h\" (UID: \"862210c8-a55c-4d92-82d2-4cfd081467aa\") " pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:37.107678 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.107645    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:37.309366 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.309338    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"]
Sep 08 16:31:37.310225 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:31:37.310205    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3839f612_8700_4b14_ba74_357592033488.slice/crio-3f13c2e7e43e3e55a42366a13671c5f9f5941e40a4135f582368c008a4fc249e WatchSource:0}: Error finding container 3f13c2e7e43e3e55a42366a13671c5f9f5941e40a4135f582368c008a4fc249e: Status 404 returned error can't find the container with id 3f13c2e7e43e3e55a42366a13671c5f9f5941e40a4135f582368c008a4fc249e
Sep 08 16:31:37.391892 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.391461    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4" event={"ID":"3839f612-8700-4b14-ba74-357592033488","Type":"ContainerStarted","Data":"b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd"}
Sep 08 16:31:37.391892 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.391525    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4" event={"ID":"3839f612-8700-4b14-ba74-357592033488","Type":"ContainerStarted","Data":"3f13c2e7e43e3e55a42366a13671c5f9f5941e40a4135f582368c008a4fc249e"}
Sep 08 16:31:37.392177 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.392077    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:37.394978 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.394956    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"]
Sep 08 16:31:37.398245 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:31:37.398222    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod862210c8_a55c_4d92_82d2_4cfd081467aa.slice/crio-5632dd7f36553fec8a98e52257b42cc99aa0b168ad7a7a20c8f9dc3cfc823307 WatchSource:0}: Error finding container 5632dd7f36553fec8a98e52257b42cc99aa0b168ad7a7a20c8f9dc3cfc823307: Status 404 returned error can't find the container with id 5632dd7f36553fec8a98e52257b42cc99aa0b168ad7a7a20c8f9dc3cfc823307
Sep 08 16:31:37.411469 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:37.411427    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4" podStartSLOduration=1.411412755 podStartE2EDuration="1.411412755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:37.410542811 +0000 UTC m=+1131.497206561" watchObservedRunningTime="2026-09-08 16:31:37.411412755 +0000 UTC m=+1131.498076503"
Sep 08 16:31:38.399193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:38.399150    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h" event={"ID":"862210c8-a55c-4d92-82d2-4cfd081467aa","Type":"ContainerStarted","Data":"6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f"}
Sep 08 16:31:38.399675 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:38.399421    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:38.399675 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:38.399469    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h" event={"ID":"862210c8-a55c-4d92-82d2-4cfd081467aa","Type":"ContainerStarted","Data":"5632dd7f36553fec8a98e52257b42cc99aa0b168ad7a7a20c8f9dc3cfc823307"}
Sep 08 16:31:38.423741 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:38.423686    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h" podStartSLOduration=2.423668165 podStartE2EDuration="2.423668165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:31:38.421281609 +0000 UTC m=+1132.507945358" watchObservedRunningTime="2026-09-08 16:31:38.423668165 +0000 UTC m=+1132.510331914"
Sep 08 16:31:49.407057 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:49.407025    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:31:50.412756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:50.412728    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:31:50.570493 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:50.570461    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"]
Sep 08 16:31:59.532854 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:59.532825    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"]
Sep 08 16:31:59.537601 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:59.537570    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"
Sep 08 16:31:59.540194 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:59.540170    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:31:59.543864 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:59.543841    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"]
Sep 08 16:31:59.557631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:59.557607    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4da4e0b-e643-48a3-82cf-a7330a3b37fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b78b587-v4857\" (UID: \"f4da4e0b-e643-48a3-82cf-a7330a3b37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"
Sep 08 16:31:59.658728 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:59.658707    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4da4e0b-e643-48a3-82cf-a7330a3b37fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b78b587-v4857\" (UID: \"f4da4e0b-e643-48a3-82cf-a7330a3b37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"
Sep 08 16:31:59.659255 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:59.659237    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4da4e0b-e643-48a3-82cf-a7330a3b37fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-1b78b587-v4857\" (UID: \"f4da4e0b-e643-48a3-82cf-a7330a3b37fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"
Sep 08 16:31:59.850176 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:31:59.850148    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"
Sep 08 16:32:00.121629 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:00.121516    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"]
Sep 08 16:32:00.125385 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:32:00.125360    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4da4e0b_e643_48a3_82cf_a7330a3b37fe.slice/crio-bad38b0322d578dd1f10842a7642b1fd462d1dc792f61122c26e511c65989652 WatchSource:0}: Error finding container bad38b0322d578dd1f10842a7642b1fd462d1dc792f61122c26e511c65989652: Status 404 returned error can't find the container with id bad38b0322d578dd1f10842a7642b1fd462d1dc792f61122c26e511c65989652
Sep 08 16:32:00.533373 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:00.533345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857" event={"ID":"f4da4e0b-e643-48a3-82cf-a7330a3b37fe","Type":"ContainerStarted","Data":"52333ae6ca5cef21326b3cb8c8882dda0f60c7b1e5f4a185239baaf372e3b3b4"}
Sep 08 16:32:00.533786 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:00.533395    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857" event={"ID":"f4da4e0b-e643-48a3-82cf-a7330a3b37fe","Type":"ContainerStarted","Data":"bad38b0322d578dd1f10842a7642b1fd462d1dc792f61122c26e511c65989652"}
Sep 08 16:32:00.559205 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:00.559159    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857" podStartSLOduration=1.559144727 podStartE2EDuration="1.559144727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:32:00.558076645 +0000 UTC m=+1154.644740393" watchObservedRunningTime="2026-09-08 16:32:00.559144727 +0000 UTC m=+1154.645808509"
Sep 08 16:32:01.542409 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:01.542378    2575 generic.go:356] "Generic (PLEG): container finished" podID="f4da4e0b-e643-48a3-82cf-a7330a3b37fe" containerID="52333ae6ca5cef21326b3cb8c8882dda0f60c7b1e5f4a185239baaf372e3b3b4" exitCode=0
Sep 08 16:32:01.542812 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:01.542418    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857" event={"ID":"f4da4e0b-e643-48a3-82cf-a7330a3b37fe","Type":"ContainerDied","Data":"52333ae6ca5cef21326b3cb8c8882dda0f60c7b1e5f4a185239baaf372e3b3b4"}
Sep 08 16:32:02.700113 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:02.700083    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"
Sep 08 16:32:02.786628 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:02.786563    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4da4e0b-e643-48a3-82cf-a7330a3b37fe-maas-api-service-ca\") pod \"f4da4e0b-e643-48a3-82cf-a7330a3b37fe\" (UID: \"f4da4e0b-e643-48a3-82cf-a7330a3b37fe\") "
Sep 08 16:32:02.786952 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:02.786928    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4da4e0b-e643-48a3-82cf-a7330a3b37fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4da4e0b-e643-48a3-82cf-a7330a3b37fe" (UID: "f4da4e0b-e643-48a3-82cf-a7330a3b37fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:32:02.888054 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:02.888010    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4da4e0b-e643-48a3-82cf-a7330a3b37fe-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:03.560879 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:03.560847    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"
Sep 08 16:32:03.561055 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:03.560885    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857" event={"ID":"f4da4e0b-e643-48a3-82cf-a7330a3b37fe","Type":"ContainerDied","Data":"bad38b0322d578dd1f10842a7642b1fd462d1dc792f61122c26e511c65989652"}
Sep 08 16:32:03.561055 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:03.560946    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bad38b0322d578dd1f10842a7642b1fd462d1dc792f61122c26e511c65989652"
Sep 08 16:32:05.590672 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.590628    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv" podUID="76b20085-9d99-4c68-91fc-24e589dfef81" containerName="main" containerID="cri-o://de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721" gracePeriod=45
Sep 08 16:32:05.855322 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.855255    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:32:05.913211 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913184    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-model-cache\") pod \"76b20085-9d99-4c68-91fc-24e589dfef81\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") "
Sep 08 16:32:05.913400 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913256    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-tmp-dir\") pod \"76b20085-9d99-4c68-91fc-24e589dfef81\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") "
Sep 08 16:32:05.913400 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913293    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b20085-9d99-4c68-91fc-24e589dfef81-tls-certs\") pod \"76b20085-9d99-4c68-91fc-24e589dfef81\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") "
Sep 08 16:32:05.913400 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913381    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcpmd\" (UniqueName: \"kubernetes.io/projected/76b20085-9d99-4c68-91fc-24e589dfef81-kube-api-access-wcpmd\") pod \"76b20085-9d99-4c68-91fc-24e589dfef81\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") "
Sep 08 16:32:05.913570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913416    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-dshm\") pod \"76b20085-9d99-4c68-91fc-24e589dfef81\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") "
Sep 08 16:32:05.913570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913456    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-home\") pod \"76b20085-9d99-4c68-91fc-24e589dfef81\" (UID: \"76b20085-9d99-4c68-91fc-24e589dfef81\") "
Sep 08 16:32:05.913570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913460    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76b20085-9d99-4c68-91fc-24e589dfef81" (UID: "76b20085-9d99-4c68-91fc-24e589dfef81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:32:05.913570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913466    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-model-cache" (OuterVolumeSpecName: "model-cache") pod "76b20085-9d99-4c68-91fc-24e589dfef81" (UID: "76b20085-9d99-4c68-91fc-24e589dfef81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:32:05.913843 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913728    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-home" (OuterVolumeSpecName: "home") pod "76b20085-9d99-4c68-91fc-24e589dfef81" (UID: "76b20085-9d99-4c68-91fc-24e589dfef81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:32:05.913843 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913749    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-tmp-dir\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:05.913843 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.913770    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-model-cache\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:05.915675 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.915649    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76b20085-9d99-4c68-91fc-24e589dfef81-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76b20085-9d99-4c68-91fc-24e589dfef81" (UID: "76b20085-9d99-4c68-91fc-24e589dfef81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:32:05.915768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.915708    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76b20085-9d99-4c68-91fc-24e589dfef81-kube-api-access-wcpmd" (OuterVolumeSpecName: "kube-api-access-wcpmd") pod "76b20085-9d99-4c68-91fc-24e589dfef81" (UID: "76b20085-9d99-4c68-91fc-24e589dfef81"). InnerVolumeSpecName "kube-api-access-wcpmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:32:05.915768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:05.915722    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-dshm" (OuterVolumeSpecName: "dshm") pod "76b20085-9d99-4c68-91fc-24e589dfef81" (UID: "76b20085-9d99-4c68-91fc-24e589dfef81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:32:06.014992 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.014959    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpmd\" (UniqueName: \"kubernetes.io/projected/76b20085-9d99-4c68-91fc-24e589dfef81-kube-api-access-wcpmd\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:06.014992 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.014986    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-dshm\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:06.014992 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.014996    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76b20085-9d99-4c68-91fc-24e589dfef81-home\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:06.015222 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.015005    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76b20085-9d99-4c68-91fc-24e589dfef81-tls-certs\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:06.579008 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.578974    2575 generic.go:356] "Generic (PLEG): container finished" podID="76b20085-9d99-4c68-91fc-24e589dfef81" containerID="de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721" exitCode=0
Sep 08 16:32:06.579187 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.579013    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"
Sep 08 16:32:06.579187 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.579011    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv" event={"ID":"76b20085-9d99-4c68-91fc-24e589dfef81","Type":"ContainerDied","Data":"de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721"}
Sep 08 16:32:06.579187 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.579068    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv" event={"ID":"76b20085-9d99-4c68-91fc-24e589dfef81","Type":"ContainerDied","Data":"92b1214e3ddb08af2a812bb3018d4b83225fb803951c897a002232eef1237ff0"}
Sep 08 16:32:06.579187 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.579091    2575 scope.go:117] "RemoveContainer" containerID="de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721"
Sep 08 16:32:06.588686 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.588659    2575 scope.go:117] "RemoveContainer" containerID="de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721"
Sep 08 16:32:06.588913 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:32:06.588893    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721\": container with ID starting with de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721 not found: ID does not exist" containerID="de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721"
Sep 08 16:32:06.588989 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.588920    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721"} err="failed to get container status \"de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721\": rpc error: code = NotFound desc = could not find container \"de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721\": container with ID starting with de70d8e0de799b01f4f3651bd0cc4f5908a75d8176079c042e7353d906d53721 not found: ID does not exist"
Sep 08 16:32:06.601380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.601351    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"]
Sep 08 16:32:06.604932 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.604907    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-ec6a8395-kserve-7864cdfbdf-kjxmv"]
Sep 08 16:32:06.966842 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.966802    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"]
Sep 08 16:32:06.967129 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.967100    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h" podUID="862210c8-a55c-4d92-82d2-4cfd081467aa" containerName="payload-processing" containerID="cri-o://6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f" gracePeriod=30
Sep 08 16:32:06.970564 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.970471    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"]
Sep 08 16:32:06.971179 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:06.970973    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4" podUID="3839f612-8700-4b14-ba74-357592033488" containerName="payload-pre-processing" containerID="cri-o://b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd" gracePeriod=30
Sep 08 16:32:07.259119 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.259091    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:32:07.261832 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.261810    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:32:07.329505 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.329461    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vf5c\" (UniqueName: \"kubernetes.io/projected/862210c8-a55c-4d92-82d2-4cfd081467aa-kube-api-access-2vf5c\") pod \"862210c8-a55c-4d92-82d2-4cfd081467aa\" (UID: \"862210c8-a55c-4d92-82d2-4cfd081467aa\") "
Sep 08 16:32:07.329697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.329512    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862210c8-a55c-4d92-82d2-4cfd081467aa-plugins-config-volume\") pod \"862210c8-a55c-4d92-82d2-4cfd081467aa\" (UID: \"862210c8-a55c-4d92-82d2-4cfd081467aa\") "
Sep 08 16:32:07.329697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.329544    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9v4d2\" (UniqueName: \"kubernetes.io/projected/3839f612-8700-4b14-ba74-357592033488-kube-api-access-9v4d2\") pod \"3839f612-8700-4b14-ba74-357592033488\" (UID: \"3839f612-8700-4b14-ba74-357592033488\") "
Sep 08 16:32:07.329697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.329611    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3839f612-8700-4b14-ba74-357592033488-plugins-config-volume\") pod \"3839f612-8700-4b14-ba74-357592033488\" (UID: \"3839f612-8700-4b14-ba74-357592033488\") "
Sep 08 16:32:07.329992 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.329966    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/862210c8-a55c-4d92-82d2-4cfd081467aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "862210c8-a55c-4d92-82d2-4cfd081467aa" (UID: "862210c8-a55c-4d92-82d2-4cfd081467aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:32:07.330302 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.330272    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3839f612-8700-4b14-ba74-357592033488-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3839f612-8700-4b14-ba74-357592033488" (UID: "3839f612-8700-4b14-ba74-357592033488"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:32:07.331768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.331740    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3839f612-8700-4b14-ba74-357592033488-kube-api-access-9v4d2" (OuterVolumeSpecName: "kube-api-access-9v4d2") pod "3839f612-8700-4b14-ba74-357592033488" (UID: "3839f612-8700-4b14-ba74-357592033488"). InnerVolumeSpecName "kube-api-access-9v4d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:32:07.332142 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.332122    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/862210c8-a55c-4d92-82d2-4cfd081467aa-kube-api-access-2vf5c" (OuterVolumeSpecName: "kube-api-access-2vf5c") pod "862210c8-a55c-4d92-82d2-4cfd081467aa" (UID: "862210c8-a55c-4d92-82d2-4cfd081467aa"). InnerVolumeSpecName "kube-api-access-2vf5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:32:07.355889 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.355868    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"]
Sep 08 16:32:07.362360 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.362341    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-1b78b587-v4857"]
Sep 08 16:32:07.431075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.431021    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/862210c8-a55c-4d92-82d2-4cfd081467aa-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:07.431075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.431046    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v4d2\" (UniqueName: \"kubernetes.io/projected/3839f612-8700-4b14-ba74-357592033488-kube-api-access-9v4d2\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:07.431075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.431061    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3839f612-8700-4b14-ba74-357592033488-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:07.431075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.431074    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vf5c\" (UniqueName: \"kubernetes.io/projected/862210c8-a55c-4d92-82d2-4cfd081467aa-kube-api-access-2vf5c\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:32:07.588133 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.588098    2575 generic.go:356] "Generic (PLEG): container finished" podID="3839f612-8700-4b14-ba74-357592033488" containerID="b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd" exitCode=0
Sep 08 16:32:07.588278 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.588139    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"
Sep 08 16:32:07.588278 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.588171    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4" event={"ID":"3839f612-8700-4b14-ba74-357592033488","Type":"ContainerDied","Data":"b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd"}
Sep 08 16:32:07.588278 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.588217    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4" event={"ID":"3839f612-8700-4b14-ba74-357592033488","Type":"ContainerDied","Data":"3f13c2e7e43e3e55a42366a13671c5f9f5941e40a4135f582368c008a4fc249e"}
Sep 08 16:32:07.588278 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.588238    2575 scope.go:117] "RemoveContainer" containerID="b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd"
Sep 08 16:32:07.589692 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.589669    2575 generic.go:356] "Generic (PLEG): container finished" podID="862210c8-a55c-4d92-82d2-4cfd081467aa" containerID="6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f" exitCode=0
Sep 08 16:32:07.589810 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.589704    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h" event={"ID":"862210c8-a55c-4d92-82d2-4cfd081467aa","Type":"ContainerDied","Data":"6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f"}
Sep 08 16:32:07.589810 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.589744    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h" event={"ID":"862210c8-a55c-4d92-82d2-4cfd081467aa","Type":"ContainerDied","Data":"5632dd7f36553fec8a98e52257b42cc99aa0b168ad7a7a20c8f9dc3cfc823307"}
Sep 08 16:32:07.589810 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.589706    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"
Sep 08 16:32:07.600399 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.600380    2575 scope.go:117] "RemoveContainer" containerID="b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd"
Sep 08 16:32:07.600754 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:32:07.600725    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd\": container with ID starting with b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd not found: ID does not exist" containerID="b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd"
Sep 08 16:32:07.600839 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.600766    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd"} err="failed to get container status \"b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd\": rpc error: code = NotFound desc = could not find container \"b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd\": container with ID starting with b37d94923357947bf71c35d2c89760c728c362c797414f7b621ce396773359bd not found: ID does not exist"
Sep 08 16:32:07.600839 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.600798    2575 scope.go:117] "RemoveContainer" containerID="6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f"
Sep 08 16:32:07.608789 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.608774    2575 scope.go:117] "RemoveContainer" containerID="6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f"
Sep 08 16:32:07.609032 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:32:07.608996    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f\": container with ID starting with 6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f not found: ID does not exist" containerID="6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f"
Sep 08 16:32:07.609032 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.609015    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f"} err="failed to get container status \"6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f\": rpc error: code = NotFound desc = could not find container \"6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f\": container with ID starting with 6cdcbb28882402749f51c54d2ece439aa576844ca50ad26da0cf0ab40e30a36f not found: ID does not exist"
Sep 08 16:32:07.620464 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.620442    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"]
Sep 08 16:32:07.628270 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.628246    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-1b78b587-5d75f688c4-qskh4"]
Sep 08 16:32:07.641886 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.641865    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"]
Sep 08 16:32:07.646539 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:07.646521    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-1b78b587-5d49c46f4f-47j2h"]
Sep 08 16:32:08.506800 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:08.506767    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3839f612-8700-4b14-ba74-357592033488" path="/var/lib/kubelet/pods/3839f612-8700-4b14-ba74-357592033488/volumes"
Sep 08 16:32:08.507181 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:08.507167    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76b20085-9d99-4c68-91fc-24e589dfef81" path="/var/lib/kubelet/pods/76b20085-9d99-4c68-91fc-24e589dfef81/volumes"
Sep 08 16:32:08.507478 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:08.507466    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="862210c8-a55c-4d92-82d2-4cfd081467aa" path="/var/lib/kubelet/pods/862210c8-a55c-4d92-82d2-4cfd081467aa/volumes"
Sep 08 16:32:08.507784 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:08.507770    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4da4e0b-e643-48a3-82cf-a7330a3b37fe" path="/var/lib/kubelet/pods/f4da4e0b-e643-48a3-82cf-a7330a3b37fe/volumes"
Sep 08 16:32:46.488470 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:46.488437    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:32:46.492949 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:46.492927    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:32:46.497240 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:46.497220    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:32:46.499008 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:32:46.498988    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:33:21.411313 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411275    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"]
Sep 08 16:33:21.411811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411697    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3839f612-8700-4b14-ba74-357592033488" containerName="payload-pre-processing"
Sep 08 16:33:21.411811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411708    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3839f612-8700-4b14-ba74-357592033488" containerName="payload-pre-processing"
Sep 08 16:33:21.411811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411719    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="862210c8-a55c-4d92-82d2-4cfd081467aa" containerName="payload-processing"
Sep 08 16:33:21.411811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411725    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="862210c8-a55c-4d92-82d2-4cfd081467aa" containerName="payload-processing"
Sep 08 16:33:21.411811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411735    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76b20085-9d99-4c68-91fc-24e589dfef81" containerName="main"
Sep 08 16:33:21.411811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411741    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="76b20085-9d99-4c68-91fc-24e589dfef81" containerName="main"
Sep 08 16:33:21.411811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411751    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4da4e0b-e643-48a3-82cf-a7330a3b37fe" containerName="revoke-keys"
Sep 08 16:33:21.411811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411756    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4da4e0b-e643-48a3-82cf-a7330a3b37fe" containerName="revoke-keys"
Sep 08 16:33:21.412073 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411840    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3839f612-8700-4b14-ba74-357592033488" containerName="payload-pre-processing"
Sep 08 16:33:21.412073 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411848    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="862210c8-a55c-4d92-82d2-4cfd081467aa" containerName="payload-processing"
Sep 08 16:33:21.412073 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411855    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4da4e0b-e643-48a3-82cf-a7330a3b37fe" containerName="revoke-keys"
Sep 08 16:33:21.412073 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.411862    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="76b20085-9d99-4c68-91fc-24e589dfef81" containerName="main"
Sep 08 16:33:21.414964 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.414947    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"
Sep 08 16:33:21.417481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.417452    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:33:21.424009 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.423250    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"]
Sep 08 16:33:21.565244 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.565213    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9c932a-5e39-4330-a254-87c3007a4e92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf\" (UID: \"9d9c932a-5e39-4330-a254-87c3007a4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"
Sep 08 16:33:21.666381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.666289    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9c932a-5e39-4330-a254-87c3007a4e92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf\" (UID: \"9d9c932a-5e39-4330-a254-87c3007a4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"
Sep 08 16:33:21.666946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.666922    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9c932a-5e39-4330-a254-87c3007a4e92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf\" (UID: \"9d9c932a-5e39-4330-a254-87c3007a4e92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"
Sep 08 16:33:21.729411 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.729378    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"
Sep 08 16:33:21.997889 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:21.997860    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"]
Sep 08 16:33:21.999980 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:33:21.999952    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9c932a_5e39_4330_a254_87c3007a4e92.slice/crio-8852ca418aa1a734ef40e4a87c3772c2026587f7b5cfe6885e76c818c1659de5 WatchSource:0}: Error finding container 8852ca418aa1a734ef40e4a87c3772c2026587f7b5cfe6885e76c818c1659de5: Status 404 returned error can't find the container with id 8852ca418aa1a734ef40e4a87c3772c2026587f7b5cfe6885e76c818c1659de5
Sep 08 16:33:22.996811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:22.996723    2575 generic.go:356] "Generic (PLEG): container finished" podID="9d9c932a-5e39-4330-a254-87c3007a4e92" containerID="b0192dfbdb2d53f1752ab2be0969d85f783c29f9fc74c1506f6d5181d4b3b601" exitCode=0
Sep 08 16:33:22.997200 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:22.996807    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf" event={"ID":"9d9c932a-5e39-4330-a254-87c3007a4e92","Type":"ContainerDied","Data":"b0192dfbdb2d53f1752ab2be0969d85f783c29f9fc74c1506f6d5181d4b3b601"}
Sep 08 16:33:22.997200 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:22.996854    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf" event={"ID":"9d9c932a-5e39-4330-a254-87c3007a4e92","Type":"ContainerStarted","Data":"8852ca418aa1a734ef40e4a87c3772c2026587f7b5cfe6885e76c818c1659de5"}
Sep 08 16:33:24.146684 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:24.146660    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"
Sep 08 16:33:24.287686 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:24.287607    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9c932a-5e39-4330-a254-87c3007a4e92-maas-api-service-ca\") pod \"9d9c932a-5e39-4330-a254-87c3007a4e92\" (UID: \"9d9c932a-5e39-4330-a254-87c3007a4e92\") "
Sep 08 16:33:24.287984 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:24.287964    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9c932a-5e39-4330-a254-87c3007a4e92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9d9c932a-5e39-4330-a254-87c3007a4e92" (UID: "9d9c932a-5e39-4330-a254-87c3007a4e92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:24.388777 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:24.388728    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9c932a-5e39-4330-a254-87c3007a4e92-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:25.009252 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:25.009222    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"
Sep 08 16:33:25.009451 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:25.009277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf" event={"ID":"9d9c932a-5e39-4330-a254-87c3007a4e92","Type":"ContainerDied","Data":"8852ca418aa1a734ef40e4a87c3772c2026587f7b5cfe6885e76c818c1659de5"}
Sep 08 16:33:25.009451 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:25.009318    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8852ca418aa1a734ef40e4a87c3772c2026587f7b5cfe6885e76c818c1659de5"
Sep 08 16:33:31.990705 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:31.990662    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"]
Sep 08 16:33:31.992554 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:31.992529    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0d9aa4-29814750-nfk79"]
Sep 08 16:33:32.037866 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.037829    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"]
Sep 08 16:33:32.038149 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.038117    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml" podUID="2a690817-566b-42e9-8d07-8a8691d245d5" containerName="payload-processing" containerID="cri-o://500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee" gracePeriod=30
Sep 08 16:33:32.046084 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.046042    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"]
Sep 08 16:33:32.046643 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.046290    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k" podUID="0259c31e-91c2-4573-bc93-bcc747364acb" containerName="payload-pre-processing" containerID="cri-o://113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f" gracePeriod=30
Sep 08 16:33:32.225829 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.225788    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"]
Sep 08 16:33:32.228594 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.228496    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-0d9aa4-wzdkf"]
Sep 08 16:33:32.306148 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.306124    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:33:32.335101 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.335077    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:33:32.459755 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.459715    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0259c31e-91c2-4573-bc93-bcc747364acb-plugins-config-volume\") pod \"0259c31e-91c2-4573-bc93-bcc747364acb\" (UID: \"0259c31e-91c2-4573-bc93-bcc747364acb\") "
Sep 08 16:33:32.459951 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.459765    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a690817-566b-42e9-8d07-8a8691d245d5-plugins-config-volume\") pod \"2a690817-566b-42e9-8d07-8a8691d245d5\" (UID: \"2a690817-566b-42e9-8d07-8a8691d245d5\") "
Sep 08 16:33:32.459951 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.459824    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qxqz\" (UniqueName: \"kubernetes.io/projected/0259c31e-91c2-4573-bc93-bcc747364acb-kube-api-access-8qxqz\") pod \"0259c31e-91c2-4573-bc93-bcc747364acb\" (UID: \"0259c31e-91c2-4573-bc93-bcc747364acb\") "
Sep 08 16:33:32.459951 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.459856    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwpqf\" (UniqueName: \"kubernetes.io/projected/2a690817-566b-42e9-8d07-8a8691d245d5-kube-api-access-pwpqf\") pod \"2a690817-566b-42e9-8d07-8a8691d245d5\" (UID: \"2a690817-566b-42e9-8d07-8a8691d245d5\") "
Sep 08 16:33:32.460153 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.460104    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0259c31e-91c2-4573-bc93-bcc747364acb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0259c31e-91c2-4573-bc93-bcc747364acb" (UID: "0259c31e-91c2-4573-bc93-bcc747364acb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:32.460232 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.460207    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a690817-566b-42e9-8d07-8a8691d245d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a690817-566b-42e9-8d07-8a8691d245d5" (UID: "2a690817-566b-42e9-8d07-8a8691d245d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:32.462000 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.461972    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0259c31e-91c2-4573-bc93-bcc747364acb-kube-api-access-8qxqz" (OuterVolumeSpecName: "kube-api-access-8qxqz") pod "0259c31e-91c2-4573-bc93-bcc747364acb" (UID: "0259c31e-91c2-4573-bc93-bcc747364acb"). InnerVolumeSpecName "kube-api-access-8qxqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:32.462139 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.462061    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a690817-566b-42e9-8d07-8a8691d245d5-kube-api-access-pwpqf" (OuterVolumeSpecName: "kube-api-access-pwpqf") pod "2a690817-566b-42e9-8d07-8a8691d245d5" (UID: "2a690817-566b-42e9-8d07-8a8691d245d5"). InnerVolumeSpecName "kube-api-access-pwpqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:32.506639 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.506610    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d9c932a-5e39-4330-a254-87c3007a4e92" path="/var/lib/kubelet/pods/9d9c932a-5e39-4330-a254-87c3007a4e92/volumes"
Sep 08 16:33:32.506912 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.506899    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d981481e-28b7-4303-a29e-86314a4f97f6" path="/var/lib/kubelet/pods/d981481e-28b7-4303-a29e-86314a4f97f6/volumes"
Sep 08 16:33:32.561407 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.561372    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qxqz\" (UniqueName: \"kubernetes.io/projected/0259c31e-91c2-4573-bc93-bcc747364acb-kube-api-access-8qxqz\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:32.561407 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.561403    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwpqf\" (UniqueName: \"kubernetes.io/projected/2a690817-566b-42e9-8d07-8a8691d245d5-kube-api-access-pwpqf\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:32.561407 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.561413    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0259c31e-91c2-4573-bc93-bcc747364acb-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:32.561622 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:32.561422    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a690817-566b-42e9-8d07-8a8691d245d5-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:33.063384 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.063353    2575 generic.go:356] "Generic (PLEG): container finished" podID="2a690817-566b-42e9-8d07-8a8691d245d5" containerID="500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee" exitCode=0
Sep 08 16:33:33.063811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.063410    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"
Sep 08 16:33:33.063811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.063440    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml" event={"ID":"2a690817-566b-42e9-8d07-8a8691d245d5","Type":"ContainerDied","Data":"500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee"}
Sep 08 16:33:33.063811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.063529    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml" event={"ID":"2a690817-566b-42e9-8d07-8a8691d245d5","Type":"ContainerDied","Data":"468c13fd5646d14d8a67a5cb9451fce57966d8dadbad08904eb2764cd737179f"}
Sep 08 16:33:33.063811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.063560    2575 scope.go:117] "RemoveContainer" containerID="500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee"
Sep 08 16:33:33.064995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.064837    2575 generic.go:356] "Generic (PLEG): container finished" podID="0259c31e-91c2-4573-bc93-bcc747364acb" containerID="113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f" exitCode=0
Sep 08 16:33:33.064995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.064868    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k" event={"ID":"0259c31e-91c2-4573-bc93-bcc747364acb","Type":"ContainerDied","Data":"113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f"}
Sep 08 16:33:33.064995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.064888    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"
Sep 08 16:33:33.064995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.064911    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k" event={"ID":"0259c31e-91c2-4573-bc93-bcc747364acb","Type":"ContainerDied","Data":"460a9762ba70a5a10c1fb708a431c07652207adcfbfbc901a0d506be9bb3f895"}
Sep 08 16:33:33.072832 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.072816    2575 scope.go:117] "RemoveContainer" containerID="500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee"
Sep 08 16:33:33.073125 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:33:33.073108    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee\": container with ID starting with 500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee not found: ID does not exist" containerID="500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee"
Sep 08 16:33:33.073169 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.073135    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee"} err="failed to get container status \"500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee\": rpc error: code = NotFound desc = could not find container \"500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee\": container with ID starting with 500c2bb4062ef38d87f9bdd43097a3ea07fa500282fc1be3a7d26b999ff82dee not found: ID does not exist"
Sep 08 16:33:33.073169 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.073155    2575 scope.go:117] "RemoveContainer" containerID="113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f"
Sep 08 16:33:33.080950 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.080927    2575 scope.go:117] "RemoveContainer" containerID="113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f"
Sep 08 16:33:33.081453 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:33:33.081426    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f\": container with ID starting with 113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f not found: ID does not exist" containerID="113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f"
Sep 08 16:33:33.081542 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.081458    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f"} err="failed to get container status \"113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f\": rpc error: code = NotFound desc = could not find container \"113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f\": container with ID starting with 113884f7a79ddecb88e5d3e377b49af274c10e79ea126db66e6c0674337ef05f not found: ID does not exist"
Sep 08 16:33:33.083320 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.083302    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"]
Sep 08 16:33:33.086554 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.086535    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0d9aa4-54874dbbf8-gq99k"]
Sep 08 16:33:33.096406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.096382    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"]
Sep 08 16:33:33.100744 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:33.100724    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0d9aa4-d8fb4bd87-rvmml"]
Sep 08 16:33:34.506343 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.506309    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0259c31e-91c2-4573-bc93-bcc747364acb" path="/var/lib/kubelet/pods/0259c31e-91c2-4573-bc93-bcc747364acb/volumes"
Sep 08 16:33:34.506737 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.506680    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a690817-566b-42e9-8d07-8a8691d245d5" path="/var/lib/kubelet/pods/2a690817-566b-42e9-8d07-8a8691d245d5/volumes"
Sep 08 16:33:34.971662 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.971626    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"]
Sep 08 16:33:34.972010 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.971999    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d9c932a-5e39-4330-a254-87c3007a4e92" containerName="revoke-keys"
Sep 08 16:33:34.972064 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.972012    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d9c932a-5e39-4330-a254-87c3007a4e92" containerName="revoke-keys"
Sep 08 16:33:34.972064 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.972028    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a690817-566b-42e9-8d07-8a8691d245d5" containerName="payload-processing"
Sep 08 16:33:34.972064 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.972034    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a690817-566b-42e9-8d07-8a8691d245d5" containerName="payload-processing"
Sep 08 16:33:34.972064 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.972061    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0259c31e-91c2-4573-bc93-bcc747364acb" containerName="payload-pre-processing"
Sep 08 16:33:34.972193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.972067    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0259c31e-91c2-4573-bc93-bcc747364acb" containerName="payload-pre-processing"
Sep 08 16:33:34.972193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.972125    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0259c31e-91c2-4573-bc93-bcc747364acb" containerName="payload-pre-processing"
Sep 08 16:33:34.972193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.972134    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a690817-566b-42e9-8d07-8a8691d245d5" containerName="payload-processing"
Sep 08 16:33:34.972193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.972141    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d9c932a-5e39-4330-a254-87c3007a4e92" containerName="revoke-keys"
Sep 08 16:33:34.976572 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.976553    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"
Sep 08 16:33:34.979822 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.979790    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:33:34.981534 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:34.981512    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"]
Sep 08 16:33:35.084296 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:35.084267    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef723740-5616-49d0-a49c-e15e4f531dc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-293166-c28kq\" (UID: \"ef723740-5616-49d0-a49c-e15e4f531dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"
Sep 08 16:33:35.185670 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:35.185628    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef723740-5616-49d0-a49c-e15e4f531dc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-293166-c28kq\" (UID: \"ef723740-5616-49d0-a49c-e15e4f531dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"
Sep 08 16:33:35.186255 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:35.186230    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef723740-5616-49d0-a49c-e15e4f531dc1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-293166-c28kq\" (UID: \"ef723740-5616-49d0-a49c-e15e4f531dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"
Sep 08 16:33:35.289817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:35.289761    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"
Sep 08 16:33:35.563976 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:35.563892    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"]
Sep 08 16:33:35.567021 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:33:35.566989    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef723740_5616_49d0_a49c_e15e4f531dc1.slice/crio-4359062a89187262b436221ed2a88c8f909c315025fac5c2b7eb4b8891521563 WatchSource:0}: Error finding container 4359062a89187262b436221ed2a88c8f909c315025fac5c2b7eb4b8891521563: Status 404 returned error can't find the container with id 4359062a89187262b436221ed2a88c8f909c315025fac5c2b7eb4b8891521563
Sep 08 16:33:36.084396 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:36.084305    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq" event={"ID":"ef723740-5616-49d0-a49c-e15e4f531dc1","Type":"ContainerStarted","Data":"b22fb195da90c43a5b3d0b7003d3967fcecd83b97f8ccf5808c38833e6e4c7ee"}
Sep 08 16:33:36.084396 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:36.084373    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq" event={"ID":"ef723740-5616-49d0-a49c-e15e4f531dc1","Type":"ContainerStarted","Data":"4359062a89187262b436221ed2a88c8f909c315025fac5c2b7eb4b8891521563"}
Sep 08 16:33:36.100962 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:36.100904    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq" podStartSLOduration=2.100884123 podStartE2EDuration="2.100884123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:36.100414205 +0000 UTC m=+1250.187077954" watchObservedRunningTime="2026-09-08 16:33:36.100884123 +0000 UTC m=+1250.187547882"
Sep 08 16:33:37.091802 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:37.091767    2575 generic.go:356] "Generic (PLEG): container finished" podID="ef723740-5616-49d0-a49c-e15e4f531dc1" containerID="b22fb195da90c43a5b3d0b7003d3967fcecd83b97f8ccf5808c38833e6e4c7ee" exitCode=0
Sep 08 16:33:37.092168 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:37.091837    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq" event={"ID":"ef723740-5616-49d0-a49c-e15e4f531dc1","Type":"ContainerDied","Data":"b22fb195da90c43a5b3d0b7003d3967fcecd83b97f8ccf5808c38833e6e4c7ee"}
Sep 08 16:33:38.232527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:38.232504    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"
Sep 08 16:33:38.315357 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:38.315331    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef723740-5616-49d0-a49c-e15e4f531dc1-maas-api-service-ca\") pod \"ef723740-5616-49d0-a49c-e15e4f531dc1\" (UID: \"ef723740-5616-49d0-a49c-e15e4f531dc1\") "
Sep 08 16:33:38.315694 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:38.315666    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef723740-5616-49d0-a49c-e15e4f531dc1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef723740-5616-49d0-a49c-e15e4f531dc1" (UID: "ef723740-5616-49d0-a49c-e15e4f531dc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:38.416101 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:38.416033    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef723740-5616-49d0-a49c-e15e4f531dc1-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:39.105566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.105535    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"
Sep 08 16:33:39.105734 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.105532    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq" event={"ID":"ef723740-5616-49d0-a49c-e15e4f531dc1","Type":"ContainerDied","Data":"4359062a89187262b436221ed2a88c8f909c315025fac5c2b7eb4b8891521563"}
Sep 08 16:33:39.105734 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.105636    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4359062a89187262b436221ed2a88c8f909c315025fac5c2b7eb4b8891521563"
Sep 08 16:33:39.121719 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.121691    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"]
Sep 08 16:33:39.122245 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.122230    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef723740-5616-49d0-a49c-e15e4f531dc1" containerName="revoke-keys"
Sep 08 16:33:39.122312 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.122248    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef723740-5616-49d0-a49c-e15e4f531dc1" containerName="revoke-keys"
Sep 08 16:33:39.122387 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.122376    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef723740-5616-49d0-a49c-e15e4f531dc1" containerName="revoke-keys"
Sep 08 16:33:39.125732 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.125712    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"
Sep 08 16:33:39.128781 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.128759    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:33:39.138963 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.138944    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"]
Sep 08 16:33:39.222829 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.222799    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c986776-b3e6-49a9-b56d-7b0654838e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x\" (UID: \"7c986776-b3e6-49a9-b56d-7b0654838e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"
Sep 08 16:33:39.323704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.323675    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c986776-b3e6-49a9-b56d-7b0654838e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x\" (UID: \"7c986776-b3e6-49a9-b56d-7b0654838e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"
Sep 08 16:33:39.324448 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.324424    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c986776-b3e6-49a9-b56d-7b0654838e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x\" (UID: \"7c986776-b3e6-49a9-b56d-7b0654838e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"
Sep 08 16:33:39.436700 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.436649    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"
Sep 08 16:33:39.710638 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:39.710616    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"]
Sep 08 16:33:39.712359 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:33:39.712329    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c986776_b3e6_49a9_b56d_7b0654838e30.slice/crio-d9084ae8d087f898841c868ae1ee761fd0f96b5d755ca1dfbe8e72655868f8e5 WatchSource:0}: Error finding container d9084ae8d087f898841c868ae1ee761fd0f96b5d755ca1dfbe8e72655868f8e5: Status 404 returned error can't find the container with id d9084ae8d087f898841c868ae1ee761fd0f96b5d755ca1dfbe8e72655868f8e5
Sep 08 16:33:40.112154 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:40.112121    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x" event={"ID":"7c986776-b3e6-49a9-b56d-7b0654838e30","Type":"ContainerStarted","Data":"55009650fd2bb9e3fadb7e37e7ce75c0553d9a50172769a15e75ab8931786db6"}
Sep 08 16:33:40.112290 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:40.112184    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x" event={"ID":"7c986776-b3e6-49a9-b56d-7b0654838e30","Type":"ContainerStarted","Data":"d9084ae8d087f898841c868ae1ee761fd0f96b5d755ca1dfbe8e72655868f8e5"}
Sep 08 16:33:40.127854 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:40.127812    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x" podStartSLOduration=1.127796881 podStartE2EDuration="1.127796881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:40.127337919 +0000 UTC m=+1254.214001668" watchObservedRunningTime="2026-09-08 16:33:40.127796881 +0000 UTC m=+1254.214460628"
Sep 08 16:33:41.119828 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:41.119791    2575 generic.go:356] "Generic (PLEG): container finished" podID="7c986776-b3e6-49a9-b56d-7b0654838e30" containerID="55009650fd2bb9e3fadb7e37e7ce75c0553d9a50172769a15e75ab8931786db6" exitCode=0
Sep 08 16:33:41.120216 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:41.119875    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x" event={"ID":"7c986776-b3e6-49a9-b56d-7b0654838e30","Type":"ContainerDied","Data":"55009650fd2bb9e3fadb7e37e7ce75c0553d9a50172769a15e75ab8931786db6"}
Sep 08 16:33:42.276736 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:42.276713    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"
Sep 08 16:33:42.349192 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:42.349162    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c986776-b3e6-49a9-b56d-7b0654838e30-maas-api-service-ca\") pod \"7c986776-b3e6-49a9-b56d-7b0654838e30\" (UID: \"7c986776-b3e6-49a9-b56d-7b0654838e30\") "
Sep 08 16:33:42.349477 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:42.349456    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c986776-b3e6-49a9-b56d-7b0654838e30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c986776-b3e6-49a9-b56d-7b0654838e30" (UID: "7c986776-b3e6-49a9-b56d-7b0654838e30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:42.450678 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:42.450619    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c986776-b3e6-49a9-b56d-7b0654838e30-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:43.135994 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:43.135966    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"
Sep 08 16:33:43.135994 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:43.135972    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x" event={"ID":"7c986776-b3e6-49a9-b56d-7b0654838e30","Type":"ContainerDied","Data":"d9084ae8d087f898841c868ae1ee761fd0f96b5d755ca1dfbe8e72655868f8e5"}
Sep 08 16:33:43.136202 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:43.136015    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9084ae8d087f898841c868ae1ee761fd0f96b5d755ca1dfbe8e72655868f8e5"
Sep 08 16:33:45.842959 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:45.842925    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"]
Sep 08 16:33:45.843414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:45.843148    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x" podUID="99e148e8-b6f5-44af-8a7d-c7324f5159d3" containerName="payload-pre-processing" containerID="cri-o://309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c" gracePeriod=30
Sep 08 16:33:46.087119 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.087097    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:33:46.093918 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.093870    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"]
Sep 08 16:33:46.098120 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.098095    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-293166-c28kq"]
Sep 08 16:33:46.153052 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.153023    2575 generic.go:356] "Generic (PLEG): container finished" podID="99e148e8-b6f5-44af-8a7d-c7324f5159d3" containerID="309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c" exitCode=0
Sep 08 16:33:46.153185 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.153067    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"
Sep 08 16:33:46.153185 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.153087    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x" event={"ID":"99e148e8-b6f5-44af-8a7d-c7324f5159d3","Type":"ContainerDied","Data":"309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c"}
Sep 08 16:33:46.153185 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.153138    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x" event={"ID":"99e148e8-b6f5-44af-8a7d-c7324f5159d3","Type":"ContainerDied","Data":"468b0b81be2b31798279979f062f78269723ac83d2357c583a3fb8a84ad370b9"}
Sep 08 16:33:46.153185 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.153169    2575 scope.go:117] "RemoveContainer" containerID="309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c"
Sep 08 16:33:46.162501 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.162424    2575 scope.go:117] "RemoveContainer" containerID="309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c"
Sep 08 16:33:46.162824 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:33:46.162787    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c\": container with ID starting with 309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c not found: ID does not exist" containerID="309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c"
Sep 08 16:33:46.162916 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.162835    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c"} err="failed to get container status \"309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c\": rpc error: code = NotFound desc = could not find container \"309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c\": container with ID starting with 309ed8baad014a2daf57f5c5323c1496b525e95863bd4d31e12a53c625f0a52c not found: ID does not exist"
Sep 08 16:33:46.180311 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.180288    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmnrb\" (UniqueName: \"kubernetes.io/projected/99e148e8-b6f5-44af-8a7d-c7324f5159d3-kube-api-access-pmnrb\") pod \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\" (UID: \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\") "
Sep 08 16:33:46.180627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.180414    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e148e8-b6f5-44af-8a7d-c7324f5159d3-plugins-config-volume\") pod \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\" (UID: \"99e148e8-b6f5-44af-8a7d-c7324f5159d3\") "
Sep 08 16:33:46.180930 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.180880    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e148e8-b6f5-44af-8a7d-c7324f5159d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "99e148e8-b6f5-44af-8a7d-c7324f5159d3" (UID: "99e148e8-b6f5-44af-8a7d-c7324f5159d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:46.182613 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.182588    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e148e8-b6f5-44af-8a7d-c7324f5159d3-kube-api-access-pmnrb" (OuterVolumeSpecName: "kube-api-access-pmnrb") pod "99e148e8-b6f5-44af-8a7d-c7324f5159d3" (UID: "99e148e8-b6f5-44af-8a7d-c7324f5159d3"). InnerVolumeSpecName "kube-api-access-pmnrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:46.281808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.281780    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e148e8-b6f5-44af-8a7d-c7324f5159d3-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:46.281808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.281804    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmnrb\" (UniqueName: \"kubernetes.io/projected/99e148e8-b6f5-44af-8a7d-c7324f5159d3-kube-api-access-pmnrb\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:46.474075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.474045    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"]
Sep 08 16:33:46.478102 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.478075    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-293166-947fddb8b-24b6x"]
Sep 08 16:33:46.509526 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.509491    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e148e8-b6f5-44af-8a7d-c7324f5159d3" path="/var/lib/kubelet/pods/99e148e8-b6f5-44af-8a7d-c7324f5159d3/volumes"
Sep 08 16:33:46.510059 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:46.510038    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef723740-5616-49d0-a49c-e15e4f531dc1" path="/var/lib/kubelet/pods/ef723740-5616-49d0-a49c-e15e4f531dc1/volumes"
Sep 08 16:33:49.275438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.275406    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"]
Sep 08 16:33:49.275891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.275778    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c986776-b3e6-49a9-b56d-7b0654838e30" containerName="revoke-keys"
Sep 08 16:33:49.275891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.275790    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c986776-b3e6-49a9-b56d-7b0654838e30" containerName="revoke-keys"
Sep 08 16:33:49.275891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.275806    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e148e8-b6f5-44af-8a7d-c7324f5159d3" containerName="payload-pre-processing"
Sep 08 16:33:49.275891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.275812    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e148e8-b6f5-44af-8a7d-c7324f5159d3" containerName="payload-pre-processing"
Sep 08 16:33:49.275891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.275871    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e148e8-b6f5-44af-8a7d-c7324f5159d3" containerName="payload-pre-processing"
Sep 08 16:33:49.275891 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.275882    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c986776-b3e6-49a9-b56d-7b0654838e30" containerName="revoke-keys"
Sep 08 16:33:49.280023 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.280002    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:49.282759 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.282736    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-57c06fce-dockercfg-sx49w\""
Sep 08 16:33:49.282759 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.282753    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-57c06fce\""
Sep 08 16:33:49.291000 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.290949    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"]
Sep 08 16:33:49.305110 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.305089    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a347a9-fa26-49c1-a2f4-e82f99040f61-plugins-config-volume\") pod \"payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8\" (UID: \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\") " pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:49.305200 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.305131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tft4v\" (UniqueName: \"kubernetes.io/projected/a0a347a9-fa26-49c1-a2f4-e82f99040f61-kube-api-access-tft4v\") pod \"payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8\" (UID: \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\") " pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:49.334997 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.334969    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"]
Sep 08 16:33:49.338513 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.338493    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:49.348383 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.348364    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"]
Sep 08 16:33:49.406441 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.406409    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a347a9-fa26-49c1-a2f4-e82f99040f61-plugins-config-volume\") pod \"payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8\" (UID: \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\") " pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:49.406594 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.406446    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk\" (UID: \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:49.406594 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.406495    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tft4v\" (UniqueName: \"kubernetes.io/projected/a0a347a9-fa26-49c1-a2f4-e82f99040f61-kube-api-access-tft4v\") pod \"payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8\" (UID: \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\") " pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:49.406594 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.406555    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5lj\" (UniqueName: \"kubernetes.io/projected/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-kube-api-access-nd5lj\") pod \"payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk\" (UID: \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:49.407214 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.407197    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a347a9-fa26-49c1-a2f4-e82f99040f61-plugins-config-volume\") pod \"payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8\" (UID: \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\") " pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:49.426464 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.426437    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tft4v\" (UniqueName: \"kubernetes.io/projected/a0a347a9-fa26-49c1-a2f4-e82f99040f61-kube-api-access-tft4v\") pod \"payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8\" (UID: \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\") " pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:49.507499 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.507469    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk\" (UID: \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:49.507647 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.507549    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5lj\" (UniqueName: \"kubernetes.io/projected/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-kube-api-access-nd5lj\") pod \"payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk\" (UID: \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:49.508278 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.508259    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk\" (UID: \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:49.519447 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.519412    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5lj\" (UniqueName: \"kubernetes.io/projected/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-kube-api-access-nd5lj\") pod \"payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk\" (UID: \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:49.547760 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.547694    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"]
Sep 08 16:33:49.547963 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.547925    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz" podUID="f7e88599-2cd9-422e-a851-6b82a1ddbe73" containerName="payload-processing" containerID="cri-o://7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641" gracePeriod=30
Sep 08 16:33:49.556524 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.556494    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"]
Sep 08 16:33:49.556524 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.557014    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2" podUID="7f387416-57b6-47eb-996d-d6a439e36d49" containerName="payload-pre-processing" containerID="cri-o://8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88" gracePeriod=30
Sep 08 16:33:49.591634 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.591605    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:49.651111 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.651073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:49.719670 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.719604    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"]
Sep 08 16:33:49.724911 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.724525    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b41034-2mv5x"]
Sep 08 16:33:49.863988 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.863721    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:33:49.871694 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.871670    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:33:49.873652 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.873625    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"]
Sep 08 16:33:49.875480 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:33:49.875452    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a347a9_fa26_49c1_a2f4_e82f99040f61.slice/crio-e8c3c4a3397416ee2b1e68110e6f9a7762d0c92cda613f88e7f42f25ebaf334f WatchSource:0}: Error finding container e8c3c4a3397416ee2b1e68110e6f9a7762d0c92cda613f88e7f42f25ebaf334f: Status 404 returned error can't find the container with id e8c3c4a3397416ee2b1e68110e6f9a7762d0c92cda613f88e7f42f25ebaf334f
Sep 08 16:33:49.910553 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.910528    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e88599-2cd9-422e-a851-6b82a1ddbe73-plugins-config-volume\") pod \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\" (UID: \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\") "
Sep 08 16:33:49.910683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.910565    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f387416-57b6-47eb-996d-d6a439e36d49-plugins-config-volume\") pod \"7f387416-57b6-47eb-996d-d6a439e36d49\" (UID: \"7f387416-57b6-47eb-996d-d6a439e36d49\") "
Sep 08 16:33:49.910683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.910603    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8drqd\" (UniqueName: \"kubernetes.io/projected/7f387416-57b6-47eb-996d-d6a439e36d49-kube-api-access-8drqd\") pod \"7f387416-57b6-47eb-996d-d6a439e36d49\" (UID: \"7f387416-57b6-47eb-996d-d6a439e36d49\") "
Sep 08 16:33:49.910683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.910639    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmpzn\" (UniqueName: \"kubernetes.io/projected/f7e88599-2cd9-422e-a851-6b82a1ddbe73-kube-api-access-cmpzn\") pod \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\" (UID: \"f7e88599-2cd9-422e-a851-6b82a1ddbe73\") "
Sep 08 16:33:49.911002 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.910976    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f387416-57b6-47eb-996d-d6a439e36d49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f387416-57b6-47eb-996d-d6a439e36d49" (UID: "7f387416-57b6-47eb-996d-d6a439e36d49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:49.911002 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.910981    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e88599-2cd9-422e-a851-6b82a1ddbe73-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7e88599-2cd9-422e-a851-6b82a1ddbe73" (UID: "f7e88599-2cd9-422e-a851-6b82a1ddbe73"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:33:49.912901 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.912876    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f387416-57b6-47eb-996d-d6a439e36d49-kube-api-access-8drqd" (OuterVolumeSpecName: "kube-api-access-8drqd") pod "7f387416-57b6-47eb-996d-d6a439e36d49" (UID: "7f387416-57b6-47eb-996d-d6a439e36d49"). InnerVolumeSpecName "kube-api-access-8drqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:49.913235 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.912986    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e88599-2cd9-422e-a851-6b82a1ddbe73-kube-api-access-cmpzn" (OuterVolumeSpecName: "kube-api-access-cmpzn") pod "f7e88599-2cd9-422e-a851-6b82a1ddbe73" (UID: "f7e88599-2cd9-422e-a851-6b82a1ddbe73"). InnerVolumeSpecName "kube-api-access-cmpzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:33:49.950250 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:49.950224    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"]
Sep 08 16:33:49.952002 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:33:49.951979    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06419a0_c1e9_41ac_8429_bbeeb5f76deb.slice/crio-d1d41b04417d487a56bd5051a39a17d83ecadf4ce4a86129b59e4b85a003ae8c WatchSource:0}: Error finding container d1d41b04417d487a56bd5051a39a17d83ecadf4ce4a86129b59e4b85a003ae8c: Status 404 returned error can't find the container with id d1d41b04417d487a56bd5051a39a17d83ecadf4ce4a86129b59e4b85a003ae8c
Sep 08 16:33:50.012280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.012244    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmpzn\" (UniqueName: \"kubernetes.io/projected/f7e88599-2cd9-422e-a851-6b82a1ddbe73-kube-api-access-cmpzn\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:50.012280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.012277    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e88599-2cd9-422e-a851-6b82a1ddbe73-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:50.012415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.012291    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f387416-57b6-47eb-996d-d6a439e36d49-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:50.012415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.012305    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8drqd\" (UniqueName: \"kubernetes.io/projected/7f387416-57b6-47eb-996d-d6a439e36d49-kube-api-access-8drqd\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:33:50.177575 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.177497    2575 generic.go:356] "Generic (PLEG): container finished" podID="7f387416-57b6-47eb-996d-d6a439e36d49" containerID="8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88" exitCode=0
Sep 08 16:33:50.177575 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.177554    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"
Sep 08 16:33:50.177807 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.177609    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2" event={"ID":"7f387416-57b6-47eb-996d-d6a439e36d49","Type":"ContainerDied","Data":"8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88"}
Sep 08 16:33:50.177807 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.177668    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2" event={"ID":"7f387416-57b6-47eb-996d-d6a439e36d49","Type":"ContainerDied","Data":"06b8a338bc21b0a2408ed95e9fb5870b5f3ce088cd3d7056c9d2a2e634bebe27"}
Sep 08 16:33:50.177807 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.177699    2575 scope.go:117] "RemoveContainer" containerID="8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88"
Sep 08 16:33:50.179335 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.179309    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8" event={"ID":"a0a347a9-fa26-49c1-a2f4-e82f99040f61","Type":"ContainerStarted","Data":"0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9"}
Sep 08 16:33:50.179461 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.179357    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8" event={"ID":"a0a347a9-fa26-49c1-a2f4-e82f99040f61","Type":"ContainerStarted","Data":"e8c3c4a3397416ee2b1e68110e6f9a7762d0c92cda613f88e7f42f25ebaf334f"}
Sep 08 16:33:50.179461 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.179423    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:33:50.181367 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.181343    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk" event={"ID":"e06419a0-c1e9-41ac-8429-bbeeb5f76deb","Type":"ContainerStarted","Data":"abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4"}
Sep 08 16:33:50.181465 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.181389    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk" event={"ID":"e06419a0-c1e9-41ac-8429-bbeeb5f76deb","Type":"ContainerStarted","Data":"d1d41b04417d487a56bd5051a39a17d83ecadf4ce4a86129b59e4b85a003ae8c"}
Sep 08 16:33:50.181547 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.181531    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:33:50.183420 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.183274    2575 generic.go:356] "Generic (PLEG): container finished" podID="f7e88599-2cd9-422e-a851-6b82a1ddbe73" containerID="7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641" exitCode=0
Sep 08 16:33:50.183420 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.183344    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz" event={"ID":"f7e88599-2cd9-422e-a851-6b82a1ddbe73","Type":"ContainerDied","Data":"7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641"}
Sep 08 16:33:50.183420 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.183366    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"
Sep 08 16:33:50.183420 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.183382    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz" event={"ID":"f7e88599-2cd9-422e-a851-6b82a1ddbe73","Type":"ContainerDied","Data":"e7a2cbffeb2954abf2c0288f50d6f0a8ce231b5ef7e9e30dab6cd0ceba7b3573"}
Sep 08 16:33:50.188140 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.188119    2575 scope.go:117] "RemoveContainer" containerID="8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88"
Sep 08 16:33:50.188413 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:33:50.188396    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88\": container with ID starting with 8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88 not found: ID does not exist" containerID="8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88"
Sep 08 16:33:50.188466 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.188424    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88"} err="failed to get container status \"8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88\": rpc error: code = NotFound desc = could not find container \"8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88\": container with ID starting with 8d45cf6eb61cac4595877d71d103e04b9ce95d1b31b770074b8a8b1b2d12ca88 not found: ID does not exist"
Sep 08 16:33:50.188466 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.188439    2575 scope.go:117] "RemoveContainer" containerID="7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641"
Sep 08 16:33:50.197475 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.197455    2575 scope.go:117] "RemoveContainer" containerID="7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641"
Sep 08 16:33:50.197765 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:33:50.197746    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641\": container with ID starting with 7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641 not found: ID does not exist" containerID="7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641"
Sep 08 16:33:50.197855 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.197774    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641"} err="failed to get container status \"7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641\": rpc error: code = NotFound desc = could not find container \"7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641\": container with ID starting with 7024826e1fd4cb21ea86602548ad3160474c3ba26c1d7db007bed99a5f718641 not found: ID does not exist"
Sep 08 16:33:50.201927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.201884    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk" podStartSLOduration=1.201869341 podStartE2EDuration="1.201869341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:50.200404493 +0000 UTC m=+1264.287068254" watchObservedRunningTime="2026-09-08 16:33:50.201869341 +0000 UTC m=+1264.288533087"
Sep 08 16:33:50.223704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.223670    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8" podStartSLOduration=1.223658247 podStartE2EDuration="1.223658247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:33:50.221541532 +0000 UTC m=+1264.308205281" watchObservedRunningTime="2026-09-08 16:33:50.223658247 +0000 UTC m=+1264.310321994"
Sep 08 16:33:50.236936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.236917    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"]
Sep 08 16:33:50.243804 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.243784    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b41034-8d96df4f9-rtscz"]
Sep 08 16:33:50.256541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.256518    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"]
Sep 08 16:33:50.261987 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.261962    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b41034-648d6cb56b-k4vt2"]
Sep 08 16:33:50.507009 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.506975    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c986776-b3e6-49a9-b56d-7b0654838e30" path="/var/lib/kubelet/pods/7c986776-b3e6-49a9-b56d-7b0654838e30/volumes"
Sep 08 16:33:50.507349 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.507335    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f387416-57b6-47eb-996d-d6a439e36d49" path="/var/lib/kubelet/pods/7f387416-57b6-47eb-996d-d6a439e36d49/volumes"
Sep 08 16:33:50.507697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:50.507683    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e88599-2cd9-422e-a851-6b82a1ddbe73" path="/var/lib/kubelet/pods/f7e88599-2cd9-422e-a851-6b82a1ddbe73/volumes"
Sep 08 16:33:59.197838 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.197806    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"]
Sep 08 16:33:59.198280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.198193    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e88599-2cd9-422e-a851-6b82a1ddbe73" containerName="payload-processing"
Sep 08 16:33:59.198280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.198206    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e88599-2cd9-422e-a851-6b82a1ddbe73" containerName="payload-processing"
Sep 08 16:33:59.198280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.198234    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f387416-57b6-47eb-996d-d6a439e36d49" containerName="payload-pre-processing"
Sep 08 16:33:59.198280 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.198242    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f387416-57b6-47eb-996d-d6a439e36d49" containerName="payload-pre-processing"
Sep 08 16:33:59.198418 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.198298    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e88599-2cd9-422e-a851-6b82a1ddbe73" containerName="payload-processing"
Sep 08 16:33:59.198418 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.198311    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f387416-57b6-47eb-996d-d6a439e36d49" containerName="payload-pre-processing"
Sep 08 16:33:59.202905 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.202888    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:33:59.206481 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.206455    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-ebb057-dockercfg-2xkz2\""
Sep 08 16:33:59.206686 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.206666    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-ebb057\""
Sep 08 16:33:59.217945 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.217923    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"]
Sep 08 16:33:59.280713 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.280683    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"]
Sep 08 16:33:59.284194 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.284175    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:33:59.290401 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.290342    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tftc\" (UniqueName: \"kubernetes.io/projected/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-kube-api-access-2tftc\") pod \"payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd\" (UID: \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\") " pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:33:59.290682 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.290635    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd\" (UID: \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\") " pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:33:59.298256 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.298231    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"]
Sep 08 16:33:59.391966 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.391940    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tftc\" (UniqueName: \"kubernetes.io/projected/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-kube-api-access-2tftc\") pod \"payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd\" (UID: \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\") " pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:33:59.392155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.391989    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd\" (UID: \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\") " pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:33:59.392155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.392011    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz\" (UID: \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:33:59.392155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.392063    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck9jm\" (UniqueName: \"kubernetes.io/projected/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-kube-api-access-ck9jm\") pod \"payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz\" (UID: \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:33:59.392658 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.392641    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-plugins-config-volume\") pod \"payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd\" (UID: \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\") " pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:33:59.400297 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.400280    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tftc\" (UniqueName: \"kubernetes.io/projected/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-kube-api-access-2tftc\") pod \"payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd\" (UID: \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\") " pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:33:59.492887 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.492828    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz\" (UID: \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:33:59.492887 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.492882    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck9jm\" (UniqueName: \"kubernetes.io/projected/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-kube-api-access-ck9jm\") pod \"payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz\" (UID: \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:33:59.493618 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.493573    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz\" (UID: \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:33:59.501524 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.501505    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck9jm\" (UniqueName: \"kubernetes.io/projected/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-kube-api-access-ck9jm\") pod \"payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz\" (UID: \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:33:59.513495 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.513435    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:33:59.597307 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.597278    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:33:59.785890 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.785865    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"]
Sep 08 16:33:59.787545 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:33:59.787519    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef1cf6c_79e5_4f6c_9322_d508cc9f99e8.slice/crio-82bf1ab23bb2639cd07d807c855bc9190043eac6cc9d5046c2b8083670241db7 WatchSource:0}: Error finding container 82bf1ab23bb2639cd07d807c855bc9190043eac6cc9d5046c2b8083670241db7: Status 404 returned error can't find the container with id 82bf1ab23bb2639cd07d807c855bc9190043eac6cc9d5046c2b8083670241db7
Sep 08 16:33:59.875745 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:33:59.875717    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"]
Sep 08 16:33:59.875842 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:33:59.875795    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded79dff9_84c1_4a15_97cf_7fb3db254ce7.slice/crio-531700f3d943b183b1d63088ad1e0fea9b28aa44ea4586106910a83040aa1693 WatchSource:0}: Error finding container 531700f3d943b183b1d63088ad1e0fea9b28aa44ea4586106910a83040aa1693: Status 404 returned error can't find the container with id 531700f3d943b183b1d63088ad1e0fea9b28aa44ea4586106910a83040aa1693
Sep 08 16:34:00.241189 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:00.241158    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz" event={"ID":"ed79dff9-84c1-4a15-97cf-7fb3db254ce7","Type":"ContainerStarted","Data":"7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31"}
Sep 08 16:34:00.241631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:00.241218    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz" event={"ID":"ed79dff9-84c1-4a15-97cf-7fb3db254ce7","Type":"ContainerStarted","Data":"531700f3d943b183b1d63088ad1e0fea9b28aa44ea4586106910a83040aa1693"}
Sep 08 16:34:00.241631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:00.241250    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:34:00.242832 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:00.242804    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd" event={"ID":"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8","Type":"ContainerStarted","Data":"a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d"}
Sep 08 16:34:00.242966 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:00.242846    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd" event={"ID":"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8","Type":"ContainerStarted","Data":"82bf1ab23bb2639cd07d807c855bc9190043eac6cc9d5046c2b8083670241db7"}
Sep 08 16:34:00.242966 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:00.242894    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:34:00.263078 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:00.263031    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz" podStartSLOduration=1.263018087 podStartE2EDuration="1.263018087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:00.261198994 +0000 UTC m=+1274.347862743" watchObservedRunningTime="2026-09-08 16:34:00.263018087 +0000 UTC m=+1274.349681835"
Sep 08 16:34:00.281304 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:00.281260    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd" podStartSLOduration=1.281246947 podStartE2EDuration="1.281246947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:00.279563027 +0000 UTC m=+1274.366226774" watchObservedRunningTime="2026-09-08 16:34:00.281246947 +0000 UTC m=+1274.367910696"
Sep 08 16:34:01.193255 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:01.193227    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:34:01.193437 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:01.193327    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:34:03.092822 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.092790    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"]
Sep 08 16:34:03.096525 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.096503    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:03.100196 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.100176    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-bce20b\""
Sep 08 16:34:03.111615 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.108000    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"]
Sep 08 16:34:03.228248 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.228203    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqtm\" (UniqueName: \"kubernetes.io/projected/1c720582-2ec1-453e-9ffc-fdb120bd111e-kube-api-access-pkqtm\") pod \"payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5\" (UID: \"1c720582-2ec1-453e-9ffc-fdb120bd111e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:03.228400 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.228349    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c720582-2ec1-453e-9ffc-fdb120bd111e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5\" (UID: \"1c720582-2ec1-453e-9ffc-fdb120bd111e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:03.329745 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.329703    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c720582-2ec1-453e-9ffc-fdb120bd111e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5\" (UID: \"1c720582-2ec1-453e-9ffc-fdb120bd111e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:03.329949 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.329782    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqtm\" (UniqueName: \"kubernetes.io/projected/1c720582-2ec1-453e-9ffc-fdb120bd111e-kube-api-access-pkqtm\") pod \"payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5\" (UID: \"1c720582-2ec1-453e-9ffc-fdb120bd111e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:03.330484 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.330456    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c720582-2ec1-453e-9ffc-fdb120bd111e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5\" (UID: \"1c720582-2ec1-453e-9ffc-fdb120bd111e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:03.342184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.342156    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqtm\" (UniqueName: \"kubernetes.io/projected/1c720582-2ec1-453e-9ffc-fdb120bd111e-kube-api-access-pkqtm\") pod \"payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5\" (UID: \"1c720582-2ec1-453e-9ffc-fdb120bd111e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:03.413329 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.413233    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:03.689429 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:03.689405    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"]
Sep 08 16:34:03.691048 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:34:03.691014    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c720582_2ec1_453e_9ffc_fdb120bd111e.slice/crio-8b4facf49c1c527a10f453afa5ad0b7489722b985987eeba0a9e7dcd3ae86f1e WatchSource:0}: Error finding container 8b4facf49c1c527a10f453afa5ad0b7489722b985987eeba0a9e7dcd3ae86f1e: Status 404 returned error can't find the container with id 8b4facf49c1c527a10f453afa5ad0b7489722b985987eeba0a9e7dcd3ae86f1e
Sep 08 16:34:04.270154 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:04.270123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5" event={"ID":"1c720582-2ec1-453e-9ffc-fdb120bd111e","Type":"ContainerStarted","Data":"732593dd20b76a93f422ace4c258010db45eedb53bca5d9c42c1f129b098e758"}
Sep 08 16:34:04.270620 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:04.270176    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5" event={"ID":"1c720582-2ec1-453e-9ffc-fdb120bd111e","Type":"ContainerStarted","Data":"8b4facf49c1c527a10f453afa5ad0b7489722b985987eeba0a9e7dcd3ae86f1e"}
Sep 08 16:34:04.270620 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:04.270218    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:04.287438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:04.287393    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5" podStartSLOduration=1.287356046 podStartE2EDuration="1.287356046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:04.286617372 +0000 UTC m=+1278.373281121" watchObservedRunningTime="2026-09-08 16:34:04.287356046 +0000 UTC m=+1278.374019793"
Sep 08 16:34:11.253497 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:11.253460    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:34:11.253907 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:11.253539    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:34:15.278288 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:15.278256    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:34:19.495614 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:19.495568    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"]
Sep 08 16:34:19.498994 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:19.498976    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"
Sep 08 16:34:19.501978 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:19.501960    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:34:19.508848 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:19.508824    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"]
Sep 08 16:34:19.670890 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:19.670859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da14e48f-fd05-43e8-b569-96d0794ea9ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm\" (UID: \"da14e48f-fd05-43e8-b569-96d0794ea9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"
Sep 08 16:34:19.772308 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:19.772220    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da14e48f-fd05-43e8-b569-96d0794ea9ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm\" (UID: \"da14e48f-fd05-43e8-b569-96d0794ea9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"
Sep 08 16:34:19.772837 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:19.772816    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da14e48f-fd05-43e8-b569-96d0794ea9ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm\" (UID: \"da14e48f-fd05-43e8-b569-96d0794ea9ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"
Sep 08 16:34:19.809759 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:19.809731    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"
Sep 08 16:34:20.085910 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:20.085887    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"]
Sep 08 16:34:20.088024 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:34:20.087993    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda14e48f_fd05_43e8_b569_96d0794ea9ca.slice/crio-3e9281f6141faad7aecf18bf0afd050efad0bdc42640aa2e9f8b9fb749e5e718 WatchSource:0}: Error finding container 3e9281f6141faad7aecf18bf0afd050efad0bdc42640aa2e9f8b9fb749e5e718: Status 404 returned error can't find the container with id 3e9281f6141faad7aecf18bf0afd050efad0bdc42640aa2e9f8b9fb749e5e718
Sep 08 16:34:20.369735 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:20.369652    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm" event={"ID":"da14e48f-fd05-43e8-b569-96d0794ea9ca","Type":"ContainerStarted","Data":"3e9281f6141faad7aecf18bf0afd050efad0bdc42640aa2e9f8b9fb749e5e718"}
Sep 08 16:34:21.376168 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:21.376133    2575 generic.go:356] "Generic (PLEG): container finished" podID="da14e48f-fd05-43e8-b569-96d0794ea9ca" containerID="240f7801db62034afb5c6572dd75420dca30d380f38f68801d9dfe5800903890" exitCode=0
Sep 08 16:34:21.376555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:21.376219    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm" event={"ID":"da14e48f-fd05-43e8-b569-96d0794ea9ca","Type":"ContainerDied","Data":"240f7801db62034afb5c6572dd75420dca30d380f38f68801d9dfe5800903890"}
Sep 08 16:34:22.514874 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:22.514849    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"
Sep 08 16:34:22.600153 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:22.600124    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da14e48f-fd05-43e8-b569-96d0794ea9ca-maas-api-service-ca\") pod \"da14e48f-fd05-43e8-b569-96d0794ea9ca\" (UID: \"da14e48f-fd05-43e8-b569-96d0794ea9ca\") "
Sep 08 16:34:22.600726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:22.600481    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da14e48f-fd05-43e8-b569-96d0794ea9ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da14e48f-fd05-43e8-b569-96d0794ea9ca" (UID: "da14e48f-fd05-43e8-b569-96d0794ea9ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:22.600726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:22.600678    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da14e48f-fd05-43e8-b569-96d0794ea9ca-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:23.389103 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:23.389077    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"
Sep 08 16:34:23.389269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:23.389073    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm" event={"ID":"da14e48f-fd05-43e8-b569-96d0794ea9ca","Type":"ContainerDied","Data":"3e9281f6141faad7aecf18bf0afd050efad0bdc42640aa2e9f8b9fb749e5e718"}
Sep 08 16:34:23.389269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:23.389196    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9281f6141faad7aecf18bf0afd050efad0bdc42640aa2e9f8b9fb749e5e718"
Sep 08 16:34:30.488063 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.488025    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"]
Sep 08 16:34:30.488531 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.488243    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8" podUID="a0a347a9-fa26-49c1-a2f4-e82f99040f61" containerName="payload-processing" containerID="cri-o://0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9" gracePeriod=30
Sep 08 16:34:30.522004 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.521977    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"]
Sep 08 16:34:30.522222 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.522188    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk" podUID="e06419a0-c1e9-41ac-8429-bbeeb5f76deb" containerName="payload-pre-processing" containerID="cri-o://abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4" gracePeriod=30
Sep 08 16:34:30.601874 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.601847    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"]
Sep 08 16:34:30.602251 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.602238    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da14e48f-fd05-43e8-b569-96d0794ea9ca" containerName="revoke-keys"
Sep 08 16:34:30.602298 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.602254    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="da14e48f-fd05-43e8-b569-96d0794ea9ca" containerName="revoke-keys"
Sep 08 16:34:30.602333 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.602321    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="da14e48f-fd05-43e8-b569-96d0794ea9ca" containerName="revoke-keys"
Sep 08 16:34:30.605178 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.605162    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"
Sep 08 16:34:30.608154 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.608136    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:34:30.633862 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.633834    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"]
Sep 08 16:34:30.772241 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.772177    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a362d-3cda-439f-881a-5e98b3c91be2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ebb057-24gjm\" (UID: \"0c6a362d-3cda-439f-881a-5e98b3c91be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"
Sep 08 16:34:30.817969 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.817943    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:34:30.822293 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.822270    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:34:30.872934 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.872910    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tft4v\" (UniqueName: \"kubernetes.io/projected/a0a347a9-fa26-49c1-a2f4-e82f99040f61-kube-api-access-tft4v\") pod \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\" (UID: \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\") "
Sep 08 16:34:30.873038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.872948    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a347a9-fa26-49c1-a2f4-e82f99040f61-plugins-config-volume\") pod \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\" (UID: \"a0a347a9-fa26-49c1-a2f4-e82f99040f61\") "
Sep 08 16:34:30.873038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.872993    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-plugins-config-volume\") pod \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\" (UID: \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\") "
Sep 08 16:34:30.873038 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.873009    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd5lj\" (UniqueName: \"kubernetes.io/projected/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-kube-api-access-nd5lj\") pod \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\" (UID: \"e06419a0-c1e9-41ac-8429-bbeeb5f76deb\") "
Sep 08 16:34:30.873264 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.873238    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a362d-3cda-439f-881a-5e98b3c91be2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ebb057-24gjm\" (UID: \"0c6a362d-3cda-439f-881a-5e98b3c91be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"
Sep 08 16:34:30.873394 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.873370    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a347a9-fa26-49c1-a2f4-e82f99040f61-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a0a347a9-fa26-49c1-a2f4-e82f99040f61" (UID: "a0a347a9-fa26-49c1-a2f4-e82f99040f61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:30.873445 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.873377    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e06419a0-c1e9-41ac-8429-bbeeb5f76deb" (UID: "e06419a0-c1e9-41ac-8429-bbeeb5f76deb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:30.873857 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.873839    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a362d-3cda-439f-881a-5e98b3c91be2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-ebb057-24gjm\" (UID: \"0c6a362d-3cda-439f-881a-5e98b3c91be2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"
Sep 08 16:34:30.875131 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.875106    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-kube-api-access-nd5lj" (OuterVolumeSpecName: "kube-api-access-nd5lj") pod "e06419a0-c1e9-41ac-8429-bbeeb5f76deb" (UID: "e06419a0-c1e9-41ac-8429-bbeeb5f76deb"). InnerVolumeSpecName "kube-api-access-nd5lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:30.875445 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.875423    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a347a9-fa26-49c1-a2f4-e82f99040f61-kube-api-access-tft4v" (OuterVolumeSpecName: "kube-api-access-tft4v") pod "a0a347a9-fa26-49c1-a2f4-e82f99040f61" (UID: "a0a347a9-fa26-49c1-a2f4-e82f99040f61"). InnerVolumeSpecName "kube-api-access-tft4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:30.932943 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.932917    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"
Sep 08 16:34:30.976740 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.974344    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:30.976740 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.974382    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd5lj\" (UniqueName: \"kubernetes.io/projected/e06419a0-c1e9-41ac-8429-bbeeb5f76deb-kube-api-access-nd5lj\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:30.976740 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.974397    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tft4v\" (UniqueName: \"kubernetes.io/projected/a0a347a9-fa26-49c1-a2f4-e82f99040f61-kube-api-access-tft4v\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:30.976740 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:30.974412    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0a347a9-fa26-49c1-a2f4-e82f99040f61-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:31.028131 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.028065    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"]
Sep 08 16:34:31.034468 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.034443    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-57c06fce-cs6lm"]
Sep 08 16:34:31.224456 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.224425    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"]
Sep 08 16:34:31.225344 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:34:31.225314    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c6a362d_3cda_439f_881a_5e98b3c91be2.slice/crio-48067ca37bb20707648a9df283cd79bd186a8b006613ec80f410aa85c1aec98c WatchSource:0}: Error finding container 48067ca37bb20707648a9df283cd79bd186a8b006613ec80f410aa85c1aec98c: Status 404 returned error can't find the container with id 48067ca37bb20707648a9df283cd79bd186a8b006613ec80f410aa85c1aec98c
Sep 08 16:34:31.437820 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.437791    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm" event={"ID":"0c6a362d-3cda-439f-881a-5e98b3c91be2","Type":"ContainerStarted","Data":"48067ca37bb20707648a9df283cd79bd186a8b006613ec80f410aa85c1aec98c"}
Sep 08 16:34:31.439223 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.439197    2575 generic.go:356] "Generic (PLEG): container finished" podID="a0a347a9-fa26-49c1-a2f4-e82f99040f61" containerID="0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9" exitCode=0
Sep 08 16:34:31.439329 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.439263    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"
Sep 08 16:34:31.439329 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.439287    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8" event={"ID":"a0a347a9-fa26-49c1-a2f4-e82f99040f61","Type":"ContainerDied","Data":"0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9"}
Sep 08 16:34:31.439437 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.439345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8" event={"ID":"a0a347a9-fa26-49c1-a2f4-e82f99040f61","Type":"ContainerDied","Data":"e8c3c4a3397416ee2b1e68110e6f9a7762d0c92cda613f88e7f42f25ebaf334f"}
Sep 08 16:34:31.439437 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.439364    2575 scope.go:117] "RemoveContainer" containerID="0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9"
Sep 08 16:34:31.440692 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.440671    2575 generic.go:356] "Generic (PLEG): container finished" podID="e06419a0-c1e9-41ac-8429-bbeeb5f76deb" containerID="abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4" exitCode=0
Sep 08 16:34:31.440777 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.440723    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"
Sep 08 16:34:31.440777 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.440739    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk" event={"ID":"e06419a0-c1e9-41ac-8429-bbeeb5f76deb","Type":"ContainerDied","Data":"abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4"}
Sep 08 16:34:31.440863 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.440777    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk" event={"ID":"e06419a0-c1e9-41ac-8429-bbeeb5f76deb","Type":"ContainerDied","Data":"d1d41b04417d487a56bd5051a39a17d83ecadf4ce4a86129b59e4b85a003ae8c"}
Sep 08 16:34:31.449220 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.449184    2575 scope.go:117] "RemoveContainer" containerID="0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9"
Sep 08 16:34:31.449455 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:34:31.449429    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9\": container with ID starting with 0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9 not found: ID does not exist" containerID="0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9"
Sep 08 16:34:31.449519 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.449458    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9"} err="failed to get container status \"0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9\": rpc error: code = NotFound desc = could not find container \"0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9\": container with ID starting with 0583ea2d36feb65290273f9f35670b7b5fd23afea148c829d9b0d8f1544e8ff9 not found: ID does not exist"
Sep 08 16:34:31.449519 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.449477    2575 scope.go:117] "RemoveContainer" containerID="abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4"
Sep 08 16:34:31.457663 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.457643    2575 scope.go:117] "RemoveContainer" containerID="abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4"
Sep 08 16:34:31.457911 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:34:31.457892    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4\": container with ID starting with abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4 not found: ID does not exist" containerID="abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4"
Sep 08 16:34:31.457961 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.457919    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4"} err="failed to get container status \"abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4\": rpc error: code = NotFound desc = could not find container \"abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4\": container with ID starting with abaf3df9927ea8e2ccfa51eae6f832b4bf2f54cb47d6bc20a5dfb2ef694d71d4 not found: ID does not exist"
Sep 08 16:34:31.514926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.514903    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"]
Sep 08 16:34:31.523875 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.523855    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-57c06fce-999454966-tfrnk"]
Sep 08 16:34:31.579093 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.579063    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"]
Sep 08 16:34:31.580823 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:31.580805    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-57c06fce-bbf656677-d2zd8"]
Sep 08 16:34:32.448815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:32.448777    2575 generic.go:356] "Generic (PLEG): container finished" podID="0c6a362d-3cda-439f-881a-5e98b3c91be2" containerID="5f51fba72f8b11a18c7e675db2709dc2d241bbc47bf497843e040ca4ad1c8437" exitCode=0
Sep 08 16:34:32.448997 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:32.448900    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm" event={"ID":"0c6a362d-3cda-439f-881a-5e98b3c91be2","Type":"ContainerDied","Data":"5f51fba72f8b11a18c7e675db2709dc2d241bbc47bf497843e040ca4ad1c8437"}
Sep 08 16:34:32.507830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:32.507802    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0a347a9-fa26-49c1-a2f4-e82f99040f61" path="/var/lib/kubelet/pods/a0a347a9-fa26-49c1-a2f4-e82f99040f61/volumes"
Sep 08 16:34:32.508217 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:32.508202    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da14e48f-fd05-43e8-b569-96d0794ea9ca" path="/var/lib/kubelet/pods/da14e48f-fd05-43e8-b569-96d0794ea9ca/volumes"
Sep 08 16:34:32.508529 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:32.508516    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e06419a0-c1e9-41ac-8429-bbeeb5f76deb" path="/var/lib/kubelet/pods/e06419a0-c1e9-41ac-8429-bbeeb5f76deb/volumes"
Sep 08 16:34:33.352499 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.352464    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"]
Sep 08 16:34:33.352919 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.352901    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0a347a9-fa26-49c1-a2f4-e82f99040f61" containerName="payload-processing"
Sep 08 16:34:33.352983 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.352922    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0a347a9-fa26-49c1-a2f4-e82f99040f61" containerName="payload-processing"
Sep 08 16:34:33.352983 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.352946    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e06419a0-c1e9-41ac-8429-bbeeb5f76deb" containerName="payload-pre-processing"
Sep 08 16:34:33.352983 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.352951    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e06419a0-c1e9-41ac-8429-bbeeb5f76deb" containerName="payload-pre-processing"
Sep 08 16:34:33.353109 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.353024    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0a347a9-fa26-49c1-a2f4-e82f99040f61" containerName="payload-processing"
Sep 08 16:34:33.353109 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.353034    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e06419a0-c1e9-41ac-8429-bbeeb5f76deb" containerName="payload-pre-processing"
Sep 08 16:34:33.357323 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.357303    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"]
Sep 08 16:34:33.357434 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.357418    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:33.360621 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.360605    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e22de8\""
Sep 08 16:34:33.361266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.361249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:33.369044 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.369024    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"]
Sep 08 16:34:33.373948 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.373930    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"]
Sep 08 16:34:33.400834 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.400812    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4g77\" (UniqueName: \"kubernetes.io/projected/96f84440-d65e-4968-84ed-c4360f5e468b-kube-api-access-n4g77\") pod \"payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp\" (UID: \"96f84440-d65e-4968-84ed-c4360f5e468b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:33.400955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.400845    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bgzn\" (UniqueName: \"kubernetes.io/projected/692cc1c2-4802-44c4-8444-1442a3a8e5ee-kube-api-access-7bgzn\") pod \"payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f\" (UID: \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:33.400955 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.400890    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f84440-d65e-4968-84ed-c4360f5e468b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp\" (UID: \"96f84440-d65e-4968-84ed-c4360f5e468b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:33.401039 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.400960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692cc1c2-4802-44c4-8444-1442a3a8e5ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f\" (UID: \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:33.501483 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.501448    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692cc1c2-4802-44c4-8444-1442a3a8e5ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f\" (UID: \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:33.501683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.501530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4g77\" (UniqueName: \"kubernetes.io/projected/96f84440-d65e-4968-84ed-c4360f5e468b-kube-api-access-n4g77\") pod \"payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp\" (UID: \"96f84440-d65e-4968-84ed-c4360f5e468b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:33.501683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.501565    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bgzn\" (UniqueName: \"kubernetes.io/projected/692cc1c2-4802-44c4-8444-1442a3a8e5ee-kube-api-access-7bgzn\") pod \"payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f\" (UID: \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:33.501683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.501665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f84440-d65e-4968-84ed-c4360f5e468b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp\" (UID: \"96f84440-d65e-4968-84ed-c4360f5e468b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:33.502381 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.502355    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692cc1c2-4802-44c4-8444-1442a3a8e5ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f\" (UID: \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:33.502528 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.502503    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f84440-d65e-4968-84ed-c4360f5e468b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp\" (UID: \"96f84440-d65e-4968-84ed-c4360f5e468b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:33.519451 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.519261    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bgzn\" (UniqueName: \"kubernetes.io/projected/692cc1c2-4802-44c4-8444-1442a3a8e5ee-kube-api-access-7bgzn\") pod \"payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f\" (UID: \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:33.520441 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.520394    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4g77\" (UniqueName: \"kubernetes.io/projected/96f84440-d65e-4968-84ed-c4360f5e468b-kube-api-access-n4g77\") pod \"payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp\" (UID: \"96f84440-d65e-4968-84ed-c4360f5e468b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:33.595345 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.595317    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"
Sep 08 16:34:33.669723 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.669642    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:33.677720 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.677691    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:33.703905 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.703871    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a362d-3cda-439f-881a-5e98b3c91be2-maas-api-service-ca\") pod \"0c6a362d-3cda-439f-881a-5e98b3c91be2\" (UID: \"0c6a362d-3cda-439f-881a-5e98b3c91be2\") "
Sep 08 16:34:33.704225 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.704201    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c6a362d-3cda-439f-881a-5e98b3c91be2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0c6a362d-3cda-439f-881a-5e98b3c91be2" (UID: "0c6a362d-3cda-439f-881a-5e98b3c91be2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:33.704315 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.704269    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c6a362d-3cda-439f-881a-5e98b3c91be2-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:33.947550 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.947523    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"]
Sep 08 16:34:33.949285 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:34:33.949258    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f84440_d65e_4968_84ed_c4360f5e468b.slice/crio-d0fd09eefe54ccfa714b98c415ba0288561a0238b85c29929346c23e67bb153f WatchSource:0}: Error finding container d0fd09eefe54ccfa714b98c415ba0288561a0238b85c29929346c23e67bb153f: Status 404 returned error can't find the container with id d0fd09eefe54ccfa714b98c415ba0288561a0238b85c29929346c23e67bb153f
Sep 08 16:34:33.983938 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:33.983914    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"]
Sep 08 16:34:33.985219 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:34:33.985194    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692cc1c2_4802_44c4_8444_1442a3a8e5ee.slice/crio-5f10fac1b4853e657185e6124e41cb35ded58532b839c9ea1745b7ab47018c75 WatchSource:0}: Error finding container 5f10fac1b4853e657185e6124e41cb35ded58532b839c9ea1745b7ab47018c75: Status 404 returned error can't find the container with id 5f10fac1b4853e657185e6124e41cb35ded58532b839c9ea1745b7ab47018c75
Sep 08 16:34:34.465721 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.465677    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp" event={"ID":"96f84440-d65e-4968-84ed-c4360f5e468b","Type":"ContainerStarted","Data":"7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e"}
Sep 08 16:34:34.466149 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.465734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp" event={"ID":"96f84440-d65e-4968-84ed-c4360f5e468b","Type":"ContainerStarted","Data":"d0fd09eefe54ccfa714b98c415ba0288561a0238b85c29929346c23e67bb153f"}
Sep 08 16:34:34.466149 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.465858    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:34.467122 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.467093    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"
Sep 08 16:34:34.467252 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.467118    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm" event={"ID":"0c6a362d-3cda-439f-881a-5e98b3c91be2","Type":"ContainerDied","Data":"48067ca37bb20707648a9df283cd79bd186a8b006613ec80f410aa85c1aec98c"}
Sep 08 16:34:34.467252 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.467157    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48067ca37bb20707648a9df283cd79bd186a8b006613ec80f410aa85c1aec98c"
Sep 08 16:34:34.468896 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.468869    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f" event={"ID":"692cc1c2-4802-44c4-8444-1442a3a8e5ee","Type":"ContainerStarted","Data":"4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0"}
Sep 08 16:34:34.468980 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.468921    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f" event={"ID":"692cc1c2-4802-44c4-8444-1442a3a8e5ee","Type":"ContainerStarted","Data":"5f10fac1b4853e657185e6124e41cb35ded58532b839c9ea1745b7ab47018c75"}
Sep 08 16:34:34.469022 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.468988    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:34.486055 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.486017    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp" podStartSLOduration=1.486005673 podStartE2EDuration="1.486005673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:34.484979705 +0000 UTC m=+1308.571643470" watchObservedRunningTime="2026-09-08 16:34:34.486005673 +0000 UTC m=+1308.572669421"
Sep 08 16:34:34.510009 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:34.509959    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f" podStartSLOduration=1.50990713 podStartE2EDuration="1.50990713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:34.509021043 +0000 UTC m=+1308.595684791" watchObservedRunningTime="2026-09-08 16:34:34.50990713 +0000 UTC m=+1308.596570880"
Sep 08 16:34:41.564702 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.564672    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"]
Sep 08 16:34:41.565090 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.564887    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd" podUID="eef1cf6c-79e5-4f6c-9322-d508cc9f99e8" containerName="payload-processing" containerID="cri-o://a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d" gracePeriod=30
Sep 08 16:34:41.569560 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.569378    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"]
Sep 08 16:34:41.569776 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.569636    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz" podUID="ed79dff9-84c1-4a15-97cf-7fb3db254ce7" containerName="payload-pre-processing" containerID="cri-o://7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31" gracePeriod=30
Sep 08 16:34:41.825030 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.825000    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"]
Sep 08 16:34:41.828288 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.828264    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-ebb057-24gjm"]
Sep 08 16:34:41.858338 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.858308    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:34:41.861501 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.861481    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:34:41.981536 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.981508    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-plugins-config-volume\") pod \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\" (UID: \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\") "
Sep 08 16:34:41.981711 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.981617    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-plugins-config-volume\") pod \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\" (UID: \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\") "
Sep 08 16:34:41.981711 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.981663    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ck9jm\" (UniqueName: \"kubernetes.io/projected/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-kube-api-access-ck9jm\") pod \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\" (UID: \"ed79dff9-84c1-4a15-97cf-7fb3db254ce7\") "
Sep 08 16:34:41.981711 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.981693    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2tftc\" (UniqueName: \"kubernetes.io/projected/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-kube-api-access-2tftc\") pod \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\" (UID: \"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8\") "
Sep 08 16:34:41.981962 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.981942    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eef1cf6c-79e5-4f6c-9322-d508cc9f99e8" (UID: "eef1cf6c-79e5-4f6c-9322-d508cc9f99e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:41.982013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.981942    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed79dff9-84c1-4a15-97cf-7fb3db254ce7" (UID: "ed79dff9-84c1-4a15-97cf-7fb3db254ce7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:34:41.983767 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.983737    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-kube-api-access-2tftc" (OuterVolumeSpecName: "kube-api-access-2tftc") pod "eef1cf6c-79e5-4f6c-9322-d508cc9f99e8" (UID: "eef1cf6c-79e5-4f6c-9322-d508cc9f99e8"). InnerVolumeSpecName "kube-api-access-2tftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:41.983894 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:41.983836    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-kube-api-access-ck9jm" (OuterVolumeSpecName: "kube-api-access-ck9jm") pod "ed79dff9-84c1-4a15-97cf-7fb3db254ce7" (UID: "ed79dff9-84c1-4a15-97cf-7fb3db254ce7"). InnerVolumeSpecName "kube-api-access-ck9jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:34:42.082691 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.082634    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:42.082691 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.082654    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck9jm\" (UniqueName: \"kubernetes.io/projected/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-kube-api-access-ck9jm\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:42.082691 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.082664    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tftc\" (UniqueName: \"kubernetes.io/projected/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8-kube-api-access-2tftc\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:42.082691 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.082672    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed79dff9-84c1-4a15-97cf-7fb3db254ce7-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:34:42.507519 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.507439    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c6a362d-3cda-439f-881a-5e98b3c91be2" path="/var/lib/kubelet/pods/0c6a362d-3cda-439f-881a-5e98b3c91be2/volumes"
Sep 08 16:34:42.527172 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.527140    2575 generic.go:356] "Generic (PLEG): container finished" podID="ed79dff9-84c1-4a15-97cf-7fb3db254ce7" containerID="7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31" exitCode=0
Sep 08 16:34:42.527272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.527178    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"
Sep 08 16:34:42.527272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.527216    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz" event={"ID":"ed79dff9-84c1-4a15-97cf-7fb3db254ce7","Type":"ContainerDied","Data":"7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31"}
Sep 08 16:34:42.527272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.527267    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz" event={"ID":"ed79dff9-84c1-4a15-97cf-7fb3db254ce7","Type":"ContainerDied","Data":"531700f3d943b183b1d63088ad1e0fea9b28aa44ea4586106910a83040aa1693"}
Sep 08 16:34:42.527435 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.527288    2575 scope.go:117] "RemoveContainer" containerID="7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31"
Sep 08 16:34:42.528708 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.528687    2575 generic.go:356] "Generic (PLEG): container finished" podID="eef1cf6c-79e5-4f6c-9322-d508cc9f99e8" containerID="a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d" exitCode=0
Sep 08 16:34:42.528785 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.528721    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd" event={"ID":"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8","Type":"ContainerDied","Data":"a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d"}
Sep 08 16:34:42.528785 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.528753    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"
Sep 08 16:34:42.528954 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.528756    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd" event={"ID":"eef1cf6c-79e5-4f6c-9322-d508cc9f99e8","Type":"ContainerDied","Data":"82bf1ab23bb2639cd07d807c855bc9190043eac6cc9d5046c2b8083670241db7"}
Sep 08 16:34:42.537326 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.537309    2575 scope.go:117] "RemoveContainer" containerID="7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31"
Sep 08 16:34:42.537619 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:34:42.537569    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31\": container with ID starting with 7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31 not found: ID does not exist" containerID="7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31"
Sep 08 16:34:42.537683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.537626    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31"} err="failed to get container status \"7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31\": rpc error: code = NotFound desc = could not find container \"7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31\": container with ID starting with 7cc58f9b531d50f2111507dabfab06348d7e2c3dbcd82db71a8ac85ab209fd31 not found: ID does not exist"
Sep 08 16:34:42.537683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.537642    2575 scope.go:117] "RemoveContainer" containerID="a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d"
Sep 08 16:34:42.545892 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.545876    2575 scope.go:117] "RemoveContainer" containerID="a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d"
Sep 08 16:34:42.546138 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:34:42.546120    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d\": container with ID starting with a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d not found: ID does not exist" containerID="a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d"
Sep 08 16:34:42.546193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.546143    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d"} err="failed to get container status \"a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d\": rpc error: code = NotFound desc = could not find container \"a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d\": container with ID starting with a752869a5da2080be9700dc8f7a5515ced61c303babe5f503c819ee8b11ece9d not found: ID does not exist"
Sep 08 16:34:42.546849 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.546829    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"]
Sep 08 16:34:42.549858 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.549840    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-ebb057-794bb758d9-4w2pd"]
Sep 08 16:34:42.564105 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.564082    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"]
Sep 08 16:34:42.568424 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:42.568405    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-ebb057-b5c684cf4-679bz"]
Sep 08 16:34:44.507696 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:44.507667    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed79dff9-84c1-4a15-97cf-7fb3db254ce7" path="/var/lib/kubelet/pods/ed79dff9-84c1-4a15-97cf-7fb3db254ce7/volumes"
Sep 08 16:34:44.508041 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:44.508008    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef1cf6c-79e5-4f6c-9322-d508cc9f99e8" path="/var/lib/kubelet/pods/eef1cf6c-79e5-4f6c-9322-d508cc9f99e8/volumes"
Sep 08 16:34:45.479573 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:45.479539    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:34:45.479744 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:45.479661    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:34:58.828666 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.828630    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"]
Sep 08 16:34:58.829015 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829006    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed79dff9-84c1-4a15-97cf-7fb3db254ce7" containerName="payload-pre-processing"
Sep 08 16:34:58.829068 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829017    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed79dff9-84c1-4a15-97cf-7fb3db254ce7" containerName="payload-pre-processing"
Sep 08 16:34:58.829068 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829033    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c6a362d-3cda-439f-881a-5e98b3c91be2" containerName="revoke-keys"
Sep 08 16:34:58.829068 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829038    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c6a362d-3cda-439f-881a-5e98b3c91be2" containerName="revoke-keys"
Sep 08 16:34:58.829068 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829048    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef1cf6c-79e5-4f6c-9322-d508cc9f99e8" containerName="payload-processing"
Sep 08 16:34:58.829068 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829054    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef1cf6c-79e5-4f6c-9322-d508cc9f99e8" containerName="payload-processing"
Sep 08 16:34:58.829215 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829119    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c6a362d-3cda-439f-881a-5e98b3c91be2" containerName="revoke-keys"
Sep 08 16:34:58.829215 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829128    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef1cf6c-79e5-4f6c-9322-d508cc9f99e8" containerName="payload-processing"
Sep 08 16:34:58.829215 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.829137    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed79dff9-84c1-4a15-97cf-7fb3db254ce7" containerName="payload-pre-processing"
Sep 08 16:34:58.833884 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.833864    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:58.837757 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.837741    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7a2cf0\"/\"default-dockercfg-8bvfd\""
Sep 08 16:34:58.837847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.837810    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7a2cf0\"/\"kube-root-ca.crt\""
Sep 08 16:34:58.837847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.837835    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-7a2cf0\"/\"noten-model-7a2cf0-kserve-self-signed-certs\""
Sep 08 16:34:58.837954 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.837810    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-7a2cf0\"/\"openshift-service-ca.crt\""
Sep 08 16:34:58.845405 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.845383    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"]
Sep 08 16:34:58.921793 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.921766    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-dshm\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:58.921886 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.921797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:58.921886 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.921819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-home\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:58.921886 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.921835    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-model-cache\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:58.921886 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.921859    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnpmr\" (UniqueName: \"kubernetes.io/projected/6c88f966-8594-4176-89ca-a1e5f640f80e-kube-api-access-xnpmr\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:58.922044 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:58.921910    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-tmp-dir\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.022940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.022912    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-tmp-dir\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.023115 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.023021    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-dshm\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.023115 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.023048    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.023115 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.023078    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-home\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.023115 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.023098    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-model-cache\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.023312 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.023129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnpmr\" (UniqueName: \"kubernetes.io/projected/6c88f966-8594-4176-89ca-a1e5f640f80e-kube-api-access-xnpmr\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.023366 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.023332    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-tmp-dir\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.023759 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.023736    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-home\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.023947 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.023920    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-model-cache\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.026127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.026101    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-dshm\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.026406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.026387    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.050314 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.050289    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnpmr\" (UniqueName: \"kubernetes.io/projected/6c88f966-8594-4176-89ca-a1e5f640f80e-kube-api-access-xnpmr\") pod \"noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") " pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.142428 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.142369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:34:59.449828 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.449803    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"]
Sep 08 16:34:59.451846 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:34:59.451819    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c88f966_8594_4176_89ca_a1e5f640f80e.slice/crio-48651456fd552056fee245ec87673ec13ffc48499f5b6038fd5880998527c080 WatchSource:0}: Error finding container 48651456fd552056fee245ec87673ec13ffc48499f5b6038fd5880998527c080: Status 404 returned error can't find the container with id 48651456fd552056fee245ec87673ec13ffc48499f5b6038fd5880998527c080
Sep 08 16:34:59.629619 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.629552    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" event={"ID":"6c88f966-8594-4176-89ca-a1e5f640f80e","Type":"ContainerStarted","Data":"443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f"}
Sep 08 16:34:59.629776 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.629634    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" event={"ID":"6c88f966-8594-4176-89ca-a1e5f640f80e","Type":"ContainerStarted","Data":"48651456fd552056fee245ec87673ec13ffc48499f5b6038fd5880998527c080"}
Sep 08 16:34:59.660303 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:34:59.660241    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" podStartSLOduration=1.66022164 podStartE2EDuration="1.66022164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:34:59.658049796 +0000 UTC m=+1333.744713545" watchObservedRunningTime="2026-09-08 16:34:59.66022164 +0000 UTC m=+1333.746885390"
Sep 08 16:35:00.566951 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.566917    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"]
Sep 08 16:35:00.570720 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.570698    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:00.576141 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.576125    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-8ac343-dockercfg-vplwm\""
Sep 08 16:35:00.577392 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.577362    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-8ac343\""
Sep 08 16:35:00.612618 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.612569    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"]
Sep 08 16:35:00.703704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.703675    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"]
Sep 08 16:35:00.707115 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.707096    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:00.709817 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:00.709792    2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh\" is forbidden: User \"system:node:ip-10-0-128-92.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-128-92.ec2.internal' and this object" podUID="a44117c9-b416-4f48-99bb-2f4a2c264102" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:00.723383 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.723364    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"]
Sep 08 16:35:00.744388 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.744366    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vft42\" (UniqueName: \"kubernetes.io/projected/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-kube-api-access-vft42\") pod \"payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6\" (UID: \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:00.744499 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.744430    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6\" (UID: \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:00.845798 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.845770    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vft42\" (UniqueName: \"kubernetes.io/projected/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-kube-api-access-vft42\") pod \"payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6\" (UID: \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:00.845908 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.845824    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6\" (UID: \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:00.845908 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.845868    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk895\" (UniqueName: \"kubernetes.io/projected/a44117c9-b416-4f48-99bb-2f4a2c264102-kube-api-access-kk895\") pod \"payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh\" (UID: \"a44117c9-b416-4f48-99bb-2f4a2c264102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:00.845908 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.845887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44117c9-b416-4f48-99bb-2f4a2c264102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh\" (UID: \"a44117c9-b416-4f48-99bb-2f4a2c264102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:00.846519 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.846502    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6\" (UID: \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:00.859609 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.859568    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vft42\" (UniqueName: \"kubernetes.io/projected/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-kube-api-access-vft42\") pod \"payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6\" (UID: \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:00.881559 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.881538    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:00.947316 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.947285    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk895\" (UniqueName: \"kubernetes.io/projected/a44117c9-b416-4f48-99bb-2f4a2c264102-kube-api-access-kk895\") pod \"payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh\" (UID: \"a44117c9-b416-4f48-99bb-2f4a2c264102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:00.947439 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.947331    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44117c9-b416-4f48-99bb-2f4a2c264102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh\" (UID: \"a44117c9-b416-4f48-99bb-2f4a2c264102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:00.948104 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.948083    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44117c9-b416-4f48-99bb-2f4a2c264102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh\" (UID: \"a44117c9-b416-4f48-99bb-2f4a2c264102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:00.960960 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:00.960934    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk895\" (UniqueName: \"kubernetes.io/projected/a44117c9-b416-4f48-99bb-2f4a2c264102-kube-api-access-kk895\") pod \"payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh\" (UID: \"a44117c9-b416-4f48-99bb-2f4a2c264102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:01.017088 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.017064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:01.167937 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.167840    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"]
Sep 08 16:35:01.170158 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:35:01.170127    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a36029_ccde_4a5f_8ce9_a909c4ed8e85.slice/crio-d431c0a31b94fa36fea8833e47d3b6b72c9478e1ff8ce8503b9bb7a7e71f85c1 WatchSource:0}: Error finding container d431c0a31b94fa36fea8833e47d3b6b72c9478e1ff8ce8503b9bb7a7e71f85c1: Status 404 returned error can't find the container with id d431c0a31b94fa36fea8833e47d3b6b72c9478e1ff8ce8503b9bb7a7e71f85c1
Sep 08 16:35:01.303875 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.303721    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"]
Sep 08 16:35:01.306441 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:35:01.306414    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44117c9_b416_4f48_99bb_2f4a2c264102.slice/crio-fa9f44710d6c98505a51d962c2a91c0418bf0fd3aeb071939380fe6491661a58 WatchSource:0}: Error finding container fa9f44710d6c98505a51d962c2a91c0418bf0fd3aeb071939380fe6491661a58: Status 404 returned error can't find the container with id fa9f44710d6c98505a51d962c2a91c0418bf0fd3aeb071939380fe6491661a58
Sep 08 16:35:01.654008 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.653839    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6" event={"ID":"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85","Type":"ContainerStarted","Data":"2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b"}
Sep 08 16:35:01.654008 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.653936    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:01.654008 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.653962    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6" event={"ID":"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85","Type":"ContainerStarted","Data":"d431c0a31b94fa36fea8833e47d3b6b72c9478e1ff8ce8503b9bb7a7e71f85c1"}
Sep 08 16:35:01.655756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.655731    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh" event={"ID":"a44117c9-b416-4f48-99bb-2f4a2c264102","Type":"ContainerStarted","Data":"6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8"}
Sep 08 16:35:01.655880 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.655770    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh" event={"ID":"a44117c9-b416-4f48-99bb-2f4a2c264102","Type":"ContainerStarted","Data":"fa9f44710d6c98505a51d962c2a91c0418bf0fd3aeb071939380fe6491661a58"}
Sep 08 16:35:01.655880 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.655789    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:01.695064 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.695021    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6" podStartSLOduration=1.695008759 podStartE2EDuration="1.695008759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:01.693284207 +0000 UTC m=+1335.779947958" watchObservedRunningTime="2026-09-08 16:35:01.695008759 +0000 UTC m=+1335.781672506"
Sep 08 16:35:01.764629 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:01.764567    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh" podStartSLOduration=1.764554417 podStartE2EDuration="1.764554417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:01.762054407 +0000 UTC m=+1335.848718156" watchObservedRunningTime="2026-09-08 16:35:01.764554417 +0000 UTC m=+1335.851218222"
Sep 08 16:35:09.142570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:09.142498    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:35:09.142570 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:09.142550    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:35:09.154440 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:09.154415    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:35:09.712071 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:09.712035    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:35:10.769993 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:10.769951    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"]
Sep 08 16:35:10.839644 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:10.839612    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-self-signed-certs: secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:10.839810 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:10.839690    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs podName:6c88f966-8594-4176-89ca-a1e5f640f80e nodeName:}" failed. No retries permitted until 2026-09-08 16:35:11.339670942 +0000 UTC m=+1345.426334668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs") pod "noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e") : secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:11.343491 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:11.343452    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-self-signed-certs: secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:11.343698 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:11.343538    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs podName:6c88f966-8594-4176-89ca-a1e5f640f80e nodeName:}" failed. No retries permitted until 2026-09-08 16:35:12.343520701 +0000 UTC m=+1346.430184442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs") pod "noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e") : secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:12.351293 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:12.351254    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-self-signed-certs: secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:12.351774 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:12.351341    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs podName:6c88f966-8594-4176-89ca-a1e5f640f80e nodeName:}" failed. No retries permitted until 2026-09-08 16:35:14.351322993 +0000 UTC m=+1348.437986733 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs") pod "noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e") : secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:12.664503 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:12.664418    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:35:12.664503 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:12.664498    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:35:14.366908 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:14.366875    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-self-signed-certs: secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:14.367260 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:14.366941    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs podName:6c88f966-8594-4176-89ca-a1e5f640f80e nodeName:}" failed. No retries permitted until 2026-09-08 16:35:18.366926535 +0000 UTC m=+1352.453590261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs") pod "noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e") : secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:18.404465 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:18.404428    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-self-signed-certs: secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:18.404978 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:18.404521    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs podName:6c88f966-8594-4176-89ca-a1e5f640f80e nodeName:}" failed. No retries permitted until 2026-09-08 16:35:26.404505474 +0000 UTC m=+1360.491169200 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs") pod "noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e") : secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:26.482662 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:26.482620    2575 secret.go:189] Couldn't get secret ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-self-signed-certs: secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:26.483077 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:26.482697    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs podName:6c88f966-8594-4176-89ca-a1e5f640f80e nodeName:}" failed. No retries permitted until 2026-09-08 16:35:42.482681828 +0000 UTC m=+1376.569345554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs") pod "noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e") : secret "noten-model-7a2cf0-kserve-self-signed-certs" not found
Sep 08 16:35:26.737302 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:26.737216    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" podUID="6c88f966-8594-4176-89ca-a1e5f640f80e" containerName="main" containerID="cri-o://443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f" gracePeriod=45
Sep 08 16:35:26.996500 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:26.996439    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:35:27.087545 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087514    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnpmr\" (UniqueName: \"kubernetes.io/projected/6c88f966-8594-4176-89ca-a1e5f640f80e-kube-api-access-xnpmr\") pod \"6c88f966-8594-4176-89ca-a1e5f640f80e\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") "
Sep 08 16:35:27.087726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087563    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-model-cache\") pod \"6c88f966-8594-4176-89ca-a1e5f640f80e\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") "
Sep 08 16:35:27.087726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087594    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-tmp-dir\") pod \"6c88f966-8594-4176-89ca-a1e5f640f80e\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") "
Sep 08 16:35:27.087726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087663    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-dshm\") pod \"6c88f966-8594-4176-89ca-a1e5f640f80e\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") "
Sep 08 16:35:27.087726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087684    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-home\") pod \"6c88f966-8594-4176-89ca-a1e5f640f80e\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") "
Sep 08 16:35:27.087936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087728    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs\") pod \"6c88f966-8594-4176-89ca-a1e5f640f80e\" (UID: \"6c88f966-8594-4176-89ca-a1e5f640f80e\") "
Sep 08 16:35:27.087936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087813    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c88f966-8594-4176-89ca-a1e5f640f80e" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:35:27.087936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087909    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c88f966-8594-4176-89ca-a1e5f640f80e" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:35:27.088103 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.087988    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-home" (OuterVolumeSpecName: "home") pod "6c88f966-8594-4176-89ca-a1e5f640f80e" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:35:27.088158 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.088098    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-model-cache\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:35:27.088158 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.088118    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-tmp-dir\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:35:27.088158 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.088132    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-home\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:35:27.089830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.089807    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c88f966-8594-4176-89ca-a1e5f640f80e" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:35:27.089946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.089834    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-dshm" (OuterVolumeSpecName: "dshm") pod "6c88f966-8594-4176-89ca-a1e5f640f80e" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:35:27.089946 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.089908    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c88f966-8594-4176-89ca-a1e5f640f80e-kube-api-access-xnpmr" (OuterVolumeSpecName: "kube-api-access-xnpmr") pod "6c88f966-8594-4176-89ca-a1e5f640f80e" (UID: "6c88f966-8594-4176-89ca-a1e5f640f80e"). InnerVolumeSpecName "kube-api-access-xnpmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:35:27.188715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.188671    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c88f966-8594-4176-89ca-a1e5f640f80e-dshm\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:35:27.188715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.188708    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c88f966-8594-4176-89ca-a1e5f640f80e-tls-certs\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:35:27.188715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.188720    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnpmr\" (UniqueName: \"kubernetes.io/projected/6c88f966-8594-4176-89ca-a1e5f640f80e-kube-api-access-xnpmr\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:35:27.802313 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.802275    2575 generic.go:356] "Generic (PLEG): container finished" podID="6c88f966-8594-4176-89ca-a1e5f640f80e" containerID="443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f" exitCode=0
Sep 08 16:35:27.802768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.802323    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"
Sep 08 16:35:27.802768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.802362    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" event={"ID":"6c88f966-8594-4176-89ca-a1e5f640f80e","Type":"ContainerDied","Data":"443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f"}
Sep 08 16:35:27.802768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.802412    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw" event={"ID":"6c88f966-8594-4176-89ca-a1e5f640f80e","Type":"ContainerDied","Data":"48651456fd552056fee245ec87673ec13ffc48499f5b6038fd5880998527c080"}
Sep 08 16:35:27.802768 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.802430    2575 scope.go:117] "RemoveContainer" containerID="443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f"
Sep 08 16:35:27.812139 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.812118    2575 scope.go:117] "RemoveContainer" containerID="443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f"
Sep 08 16:35:27.812395 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:35:27.812377    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f\": container with ID starting with 443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f not found: ID does not exist" containerID="443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f"
Sep 08 16:35:27.812443 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.812403    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f"} err="failed to get container status \"443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f\": rpc error: code = NotFound desc = could not find container \"443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f\": container with ID starting with 443f464375ce259195b017c052e41a902ff1c7728137f18b5400e27a3f427f8f not found: ID does not exist"
Sep 08 16:35:27.829348 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.829326    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"]
Sep 08 16:35:27.836201 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:27.836181    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-7a2cf0/noten-model-7a2cf0-kserve-7fc4cf8ddc-5z2dw"]
Sep 08 16:35:28.507304 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.507268    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c88f966-8594-4176-89ca-a1e5f640f80e" path="/var/lib/kubelet/pods/6c88f966-8594-4176-89ca-a1e5f640f80e/volumes"
Sep 08 16:35:28.756242 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.756207    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"]
Sep 08 16:35:28.756636 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.756623    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c88f966-8594-4176-89ca-a1e5f640f80e" containerName="main"
Sep 08 16:35:28.756636 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.756637    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c88f966-8594-4176-89ca-a1e5f640f80e" containerName="main"
Sep 08 16:35:28.756730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.756709    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c88f966-8594-4176-89ca-a1e5f640f80e" containerName="main"
Sep 08 16:35:28.761428 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.761372    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:28.767405 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.767384    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-8ded33\""
Sep 08 16:35:28.777821 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.777797    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"]
Sep 08 16:35:28.803214 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.803186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tshw\" (UniqueName: \"kubernetes.io/projected/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-kube-api-access-5tshw\") pod \"payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h\" (UID: \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:28.803562 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.803279    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h\" (UID: \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:28.829810 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.829783    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"]
Sep 08 16:35:28.833207 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.833192    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:28.835705 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.835682    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-8ded33-dockercfg-vndcd\""
Sep 08 16:35:28.844518 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.844497    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"]
Sep 08 16:35:28.904060 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.904022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tshw\" (UniqueName: \"kubernetes.io/projected/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-kube-api-access-5tshw\") pod \"payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h\" (UID: \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:28.904226 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.904100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7c2c\" (UniqueName: \"kubernetes.io/projected/74cfa857-b9c7-430e-8644-ddd409fd7fe0-kube-api-access-d7c2c\") pod \"payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw\" (UID: \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:28.904226 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.904128    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cfa857-b9c7-430e-8644-ddd409fd7fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw\" (UID: \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:28.904226 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.904150    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h\" (UID: \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:28.904897 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.904877    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h\" (UID: \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:28.919445 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:28.919417    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tshw\" (UniqueName: \"kubernetes.io/projected/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-kube-api-access-5tshw\") pod \"payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h\" (UID: \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:29.004974 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.004938    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7c2c\" (UniqueName: \"kubernetes.io/projected/74cfa857-b9c7-430e-8644-ddd409fd7fe0-kube-api-access-d7c2c\") pod \"payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw\" (UID: \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:29.005131 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.004983    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cfa857-b9c7-430e-8644-ddd409fd7fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw\" (UID: \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:29.005704 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.005684    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cfa857-b9c7-430e-8644-ddd409fd7fe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw\" (UID: \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:29.014109 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.014040    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7c2c\" (UniqueName: \"kubernetes.io/projected/74cfa857-b9c7-430e-8644-ddd409fd7fe0-kube-api-access-d7c2c\") pod \"payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw\" (UID: \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:29.071698 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.071664    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:29.143900 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.143786    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:29.348419 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.348392    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"]
Sep 08 16:35:29.350258 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:35:29.350229    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af5520a_1ebe_44b3_82a1_90ef502b1dfe.slice/crio-229c62bb1cebadcd527c8c57bcfb7cb8e6c2dd20ce06fe63688c86d539cf8e7d WatchSource:0}: Error finding container 229c62bb1cebadcd527c8c57bcfb7cb8e6c2dd20ce06fe63688c86d539cf8e7d: Status 404 returned error can't find the container with id 229c62bb1cebadcd527c8c57bcfb7cb8e6c2dd20ce06fe63688c86d539cf8e7d
Sep 08 16:35:29.429343 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.429313    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"]
Sep 08 16:35:29.431075 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:35:29.431039    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74cfa857_b9c7_430e_8644_ddd409fd7fe0.slice/crio-64425c8d13e0ff51e90ca393c3d17c1d9be68409e5693d634cb5e6e2e9325495 WatchSource:0}: Error finding container 64425c8d13e0ff51e90ca393c3d17c1d9be68409e5693d634cb5e6e2e9325495: Status 404 returned error can't find the container with id 64425c8d13e0ff51e90ca393c3d17c1d9be68409e5693d634cb5e6e2e9325495
Sep 08 16:35:29.818724 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.818627    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h" event={"ID":"3af5520a-1ebe-44b3-82a1-90ef502b1dfe","Type":"ContainerStarted","Data":"410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e"}
Sep 08 16:35:29.819164 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.818794    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h" event={"ID":"3af5520a-1ebe-44b3-82a1-90ef502b1dfe","Type":"ContainerStarted","Data":"229c62bb1cebadcd527c8c57bcfb7cb8e6c2dd20ce06fe63688c86d539cf8e7d"}
Sep 08 16:35:29.821355 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.820501    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:35:29.821486 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.821385    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:29.821486 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.821425    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw" event={"ID":"74cfa857-b9c7-430e-8644-ddd409fd7fe0","Type":"ContainerStarted","Data":"50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87"}
Sep 08 16:35:29.821486 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.821452    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw" event={"ID":"74cfa857-b9c7-430e-8644-ddd409fd7fe0","Type":"ContainerStarted","Data":"64425c8d13e0ff51e90ca393c3d17c1d9be68409e5693d634cb5e6e2e9325495"}
Sep 08 16:35:29.852595 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.852544    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h" podStartSLOduration=1.852530512 podStartE2EDuration="1.852530512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:29.852058268 +0000 UTC m=+1363.938722008" watchObservedRunningTime="2026-09-08 16:35:29.852530512 +0000 UTC m=+1363.939194259"
Sep 08 16:35:29.880766 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:29.880719    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw" podStartSLOduration=1.880704536 podStartE2EDuration="1.880704536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:35:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:35:29.880088779 +0000 UTC m=+1363.966752530" watchObservedRunningTime="2026-09-08 16:35:29.880704536 +0000 UTC m=+1363.967368304"
Sep 08 16:35:40.830342 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:40.830311    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:35:41.837826 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:35:41.837794    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:36:11.836868 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.836824    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"]
Sep 08 16:36:11.841795 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.841774    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:11.849119 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.849093    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1fa1601c-dockercfg-k47p2\""
Sep 08 16:36:11.850118 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.850093    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1fa1601c\""
Sep 08 16:36:11.881503 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.881474    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2\" (UID: \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:11.881648 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.881528    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szhpb\" (UniqueName: \"kubernetes.io/projected/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-kube-api-access-szhpb\") pod \"payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2\" (UID: \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:11.900266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.900240    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"]
Sep 08 16:36:11.950894 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.950869    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"]
Sep 08 16:36:11.954402 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.954385    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:11.981964 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.981937    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2\" (UID: \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:11.982072 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.981985    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szhpb\" (UniqueName: \"kubernetes.io/projected/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-kube-api-access-szhpb\") pod \"payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2\" (UID: \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:11.982072 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.982018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9wzq\" (UniqueName: \"kubernetes.io/projected/7d01fced-12ba-4d0d-984e-fda60e20a64e-kube-api-access-f9wzq\") pod \"payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x\" (UID: \"7d01fced-12ba-4d0d-984e-fda60e20a64e\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:11.982180 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.982156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d01fced-12ba-4d0d-984e-fda60e20a64e-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x\" (UID: \"7d01fced-12ba-4d0d-984e-fda60e20a64e\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:11.982638 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:11.982621    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2\" (UID: \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:12.002078 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.002058    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szhpb\" (UniqueName: \"kubernetes.io/projected/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-kube-api-access-szhpb\") pod \"payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2\" (UID: \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:12.027771 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.027746    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"]
Sep 08 16:36:12.083503 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.083480    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9wzq\" (UniqueName: \"kubernetes.io/projected/7d01fced-12ba-4d0d-984e-fda60e20a64e-kube-api-access-f9wzq\") pod \"payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x\" (UID: \"7d01fced-12ba-4d0d-984e-fda60e20a64e\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:12.083629 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.083557    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d01fced-12ba-4d0d-984e-fda60e20a64e-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x\" (UID: \"7d01fced-12ba-4d0d-984e-fda60e20a64e\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:12.084191 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.084172    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d01fced-12ba-4d0d-984e-fda60e20a64e-plugins-config-volume\") pod \"payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x\" (UID: \"7d01fced-12ba-4d0d-984e-fda60e20a64e\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:12.098266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.098212    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9wzq\" (UniqueName: \"kubernetes.io/projected/7d01fced-12ba-4d0d-984e-fda60e20a64e-kube-api-access-f9wzq\") pod \"payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x\" (UID: \"7d01fced-12ba-4d0d-984e-fda60e20a64e\") " pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:12.152645 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.152618    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:12.266762 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.266734    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:12.446870 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.446843    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"]
Sep 08 16:36:12.449044 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:36:12.449017    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5e8ce2_ae6a_4f19_bcae_2b8bf7daf196.slice/crio-8dd783d4955a6c4d0fa66e5f53e0bc7e869750f8934f5d552f9c5a26f077ad6a WatchSource:0}: Error finding container 8dd783d4955a6c4d0fa66e5f53e0bc7e869750f8934f5d552f9c5a26f077ad6a: Status 404 returned error can't find the container with id 8dd783d4955a6c4d0fa66e5f53e0bc7e869750f8934f5d552f9c5a26f077ad6a
Sep 08 16:36:12.560054 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:12.560021    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"]
Sep 08 16:36:12.562479 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:36:12.562449    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d01fced_12ba_4d0d_984e_fda60e20a64e.slice/crio-5b77319e353611be699ecfa99aff4c1e1083111b20eeeb7bb30f9e32bed1691c WatchSource:0}: Error finding container 5b77319e353611be699ecfa99aff4c1e1083111b20eeeb7bb30f9e32bed1691c: Status 404 returned error can't find the container with id 5b77319e353611be699ecfa99aff4c1e1083111b20eeeb7bb30f9e32bed1691c
Sep 08 16:36:13.070979 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:13.070940    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2" event={"ID":"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196","Type":"ContainerStarted","Data":"702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f"}
Sep 08 16:36:13.071419 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:13.071009    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2" event={"ID":"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196","Type":"ContainerStarted","Data":"8dd783d4955a6c4d0fa66e5f53e0bc7e869750f8934f5d552f9c5a26f077ad6a"}
Sep 08 16:36:13.071419 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:13.071047    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:13.072770 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:13.072734    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x" event={"ID":"7d01fced-12ba-4d0d-984e-fda60e20a64e","Type":"ContainerStarted","Data":"48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90"}
Sep 08 16:36:13.072888 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:13.072787    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x" event={"ID":"7d01fced-12ba-4d0d-984e-fda60e20a64e","Type":"ContainerStarted","Data":"5b77319e353611be699ecfa99aff4c1e1083111b20eeeb7bb30f9e32bed1691c"}
Sep 08 16:36:13.072888 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:13.072827    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:13.107255 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:13.107205    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2" podStartSLOduration=2.107186518 podStartE2EDuration="2.107186518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:36:13.106709328 +0000 UTC m=+1407.193373079" watchObservedRunningTime="2026-09-08 16:36:13.107186518 +0000 UTC m=+1407.193850271"
Sep 08 16:36:13.144005 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:13.143960    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x" podStartSLOduration=2.143945383 podStartE2EDuration="2.143945383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:36:13.143337326 +0000 UTC m=+1407.230001085" watchObservedRunningTime="2026-09-08 16:36:13.143945383 +0000 UTC m=+1407.230609132"
Sep 08 16:36:24.080732 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:24.080687    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:36:24.081189 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:24.080765    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:36:47.210107 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:47.210076    2575 scope.go:117] "RemoveContainer" containerID="5a315bc761134a3943e44bc01b5cae8cfa9c6a2757a0a0e9b01a91d494c14529"
Sep 08 16:36:47.218611 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:47.218574    2575 scope.go:117] "RemoveContainer" containerID="19fb0bd898e0e17e42ab77f917f942533b5f71f18d130c3c957cbfdd2f5e3848"
Sep 08 16:36:47.227734 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:36:47.227713    2575 scope.go:117] "RemoveContainer" containerID="149eea2f43e72a5abd3fec301a4d2dee9f1034d9b4e38cb4320cb9f0a7a6d4f8"
Sep 08 16:37:43.952077 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:43.952034    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"]
Sep 08 16:37:43.955721 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:43.955704    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"
Sep 08 16:37:43.958419 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:43.958402    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:37:43.968602 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:43.968552    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"]
Sep 08 16:37:44.070336 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:44.070305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca198a-bed4-45b6-9d5f-49f515abeb6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6\" (UID: \"a5ca198a-bed4-45b6-9d5f-49f515abeb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"
Sep 08 16:37:44.171548 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:44.171518    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca198a-bed4-45b6-9d5f-49f515abeb6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6\" (UID: \"a5ca198a-bed4-45b6-9d5f-49f515abeb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"
Sep 08 16:37:44.172152 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:44.172128    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca198a-bed4-45b6-9d5f-49f515abeb6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6\" (UID: \"a5ca198a-bed4-45b6-9d5f-49f515abeb6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"
Sep 08 16:37:44.270272 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:44.270196    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"
Sep 08 16:37:44.552815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:44.552793    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"]
Sep 08 16:37:44.555013 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:37:44.554984    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ca198a_bed4_45b6_9d5f_49f515abeb6b.slice/crio-52643e18be5659ca9739b290929c2795dad9e4ffc85fe17824668782bd148441 WatchSource:0}: Error finding container 52643e18be5659ca9739b290929c2795dad9e4ffc85fe17824668782bd148441: Status 404 returned error can't find the container with id 52643e18be5659ca9739b290929c2795dad9e4ffc85fe17824668782bd148441
Sep 08 16:37:44.603787 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:44.603762    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6" event={"ID":"a5ca198a-bed4-45b6-9d5f-49f515abeb6b","Type":"ContainerStarted","Data":"52643e18be5659ca9739b290929c2795dad9e4ffc85fe17824668782bd148441"}
Sep 08 16:37:45.611244 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:45.611215    2575 generic.go:356] "Generic (PLEG): container finished" podID="a5ca198a-bed4-45b6-9d5f-49f515abeb6b" containerID="3a188e68910bb16dc53b42d3f232112382c66060f53e40a8f999131bf4ff367d" exitCode=0
Sep 08 16:37:45.611662 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:45.611303    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6" event={"ID":"a5ca198a-bed4-45b6-9d5f-49f515abeb6b","Type":"ContainerDied","Data":"3a188e68910bb16dc53b42d3f232112382c66060f53e40a8f999131bf4ff367d"}
Sep 08 16:37:46.528042 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:46.528012    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:37:46.531990 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:46.531969    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:37:46.534248 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:46.534228    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:37:46.538132 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:46.538114    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:37:46.768751 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:46.768716    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"
Sep 08 16:37:46.895928 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:46.895898    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca198a-bed4-45b6-9d5f-49f515abeb6b-maas-api-service-ca\") pod \"a5ca198a-bed4-45b6-9d5f-49f515abeb6b\" (UID: \"a5ca198a-bed4-45b6-9d5f-49f515abeb6b\") "
Sep 08 16:37:46.896271 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:46.896250    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ca198a-bed4-45b6-9d5f-49f515abeb6b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a5ca198a-bed4-45b6-9d5f-49f515abeb6b" (UID: "a5ca198a-bed4-45b6-9d5f-49f515abeb6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:37:46.997125 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:46.997094    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a5ca198a-bed4-45b6-9d5f-49f515abeb6b-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:37:47.254605 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:47.254509    2575 scope.go:117] "RemoveContainer" containerID="f89da7d54057e0b57fcd2bee1306c78835159d8b1ba97c0578412496ea713cff"
Sep 08 16:37:47.263326 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:47.263308    2575 scope.go:117] "RemoveContainer" containerID="e9215d9f09e128e509e4290f40d9a54fe4100b5ee02837fbb856198e273673c2"
Sep 08 16:37:47.625418 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:47.625385    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6" event={"ID":"a5ca198a-bed4-45b6-9d5f-49f515abeb6b","Type":"ContainerDied","Data":"52643e18be5659ca9739b290929c2795dad9e4ffc85fe17824668782bd148441"}
Sep 08 16:37:47.625418 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:47.625420    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"
Sep 08 16:37:47.625672 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:47.625435    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52643e18be5659ca9739b290929c2795dad9e4ffc85fe17824668782bd148441"
Sep 08 16:37:59.647491 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:59.647460    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"]
Sep 08 16:37:59.647933 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:59.647693    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x" podUID="7d01fced-12ba-4d0d-984e-fda60e20a64e" containerName="payload-processing" containerID="cri-o://48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90" gracePeriod=30
Sep 08 16:37:59.679431 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:59.679249    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"]
Sep 08 16:37:59.679556 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:59.679517    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2" podUID="1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196" containerName="payload-pre-processing" containerID="cri-o://702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f" gracePeriod=30
Sep 08 16:37:59.935840 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:59.935811    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"]
Sep 08 16:37:59.940169 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:59.940143    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1fa1601c-rd2w6"]
Sep 08 16:37:59.949048 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:59.949027    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:37:59.977155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:37:59.977132    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:38:00.001438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.000763    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-plugins-config-volume\") pod \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\" (UID: \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\") "
Sep 08 16:38:00.001438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.000817    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d01fced-12ba-4d0d-984e-fda60e20a64e-plugins-config-volume\") pod \"7d01fced-12ba-4d0d-984e-fda60e20a64e\" (UID: \"7d01fced-12ba-4d0d-984e-fda60e20a64e\") "
Sep 08 16:38:00.001438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.000889    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9wzq\" (UniqueName: \"kubernetes.io/projected/7d01fced-12ba-4d0d-984e-fda60e20a64e-kube-api-access-f9wzq\") pod \"7d01fced-12ba-4d0d-984e-fda60e20a64e\" (UID: \"7d01fced-12ba-4d0d-984e-fda60e20a64e\") "
Sep 08 16:38:00.001438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.000935    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szhpb\" (UniqueName: \"kubernetes.io/projected/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-kube-api-access-szhpb\") pod \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\" (UID: \"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196\") "
Sep 08 16:38:00.001438 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.001269    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196" (UID: "1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:38:00.001863 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.001768    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d01fced-12ba-4d0d-984e-fda60e20a64e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d01fced-12ba-4d0d-984e-fda60e20a64e" (UID: "7d01fced-12ba-4d0d-984e-fda60e20a64e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:38:00.003868 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.003841    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d01fced-12ba-4d0d-984e-fda60e20a64e-kube-api-access-f9wzq" (OuterVolumeSpecName: "kube-api-access-f9wzq") pod "7d01fced-12ba-4d0d-984e-fda60e20a64e" (UID: "7d01fced-12ba-4d0d-984e-fda60e20a64e"). InnerVolumeSpecName "kube-api-access-f9wzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:38:00.003980 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.003899    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-kube-api-access-szhpb" (OuterVolumeSpecName: "kube-api-access-szhpb") pod "1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196" (UID: "1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196"). InnerVolumeSpecName "kube-api-access-szhpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:38:00.102464 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.102440    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szhpb\" (UniqueName: \"kubernetes.io/projected/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-kube-api-access-szhpb\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.102464 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.102463    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.102612 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.102473    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d01fced-12ba-4d0d-984e-fda60e20a64e-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.102612 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.102482    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9wzq\" (UniqueName: \"kubernetes.io/projected/7d01fced-12ba-4d0d-984e-fda60e20a64e-kube-api-access-f9wzq\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:38:00.506957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.506924    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5ca198a-bed4-45b6-9d5f-49f515abeb6b" path="/var/lib/kubelet/pods/a5ca198a-bed4-45b6-9d5f-49f515abeb6b/volumes"
Sep 08 16:38:00.701645 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.701615    2575 generic.go:356] "Generic (PLEG): container finished" podID="7d01fced-12ba-4d0d-984e-fda60e20a64e" containerID="48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90" exitCode=0
Sep 08 16:38:00.701990 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.701660    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"
Sep 08 16:38:00.701990 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.701707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x" event={"ID":"7d01fced-12ba-4d0d-984e-fda60e20a64e","Type":"ContainerDied","Data":"48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90"}
Sep 08 16:38:00.701990 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.701762    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x" event={"ID":"7d01fced-12ba-4d0d-984e-fda60e20a64e","Type":"ContainerDied","Data":"5b77319e353611be699ecfa99aff4c1e1083111b20eeeb7bb30f9e32bed1691c"}
Sep 08 16:38:00.701990 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.701794    2575 scope.go:117] "RemoveContainer" containerID="48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90"
Sep 08 16:38:00.703193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.703172    2575 generic.go:356] "Generic (PLEG): container finished" podID="1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196" containerID="702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f" exitCode=0
Sep 08 16:38:00.703276 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.703212    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"
Sep 08 16:38:00.703343 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.703315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2" event={"ID":"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196","Type":"ContainerDied","Data":"702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f"}
Sep 08 16:38:00.703397 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.703362    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2" event={"ID":"1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196","Type":"ContainerDied","Data":"8dd783d4955a6c4d0fa66e5f53e0bc7e869750f8934f5d552f9c5a26f077ad6a"}
Sep 08 16:38:00.710753 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.710736    2575 scope.go:117] "RemoveContainer" containerID="48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90"
Sep 08 16:38:00.711013 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:38:00.710992    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90\": container with ID starting with 48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90 not found: ID does not exist" containerID="48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90"
Sep 08 16:38:00.711061 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.711020    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90"} err="failed to get container status \"48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90\": rpc error: code = NotFound desc = could not find container \"48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90\": container with ID starting with 48501daa6f13f65c2a3c43b3b27adbe3ebf7766d187b889acc90bb167cde4c90 not found: ID does not exist"
Sep 08 16:38:00.711061 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.711036    2575 scope.go:117] "RemoveContainer" containerID="702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f"
Sep 08 16:38:00.719619 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.719568    2575 scope.go:117] "RemoveContainer" containerID="702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f"
Sep 08 16:38:00.719933 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:38:00.719911    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f\": container with ID starting with 702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f not found: ID does not exist" containerID="702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f"
Sep 08 16:38:00.720004 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.719943    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f"} err="failed to get container status \"702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f\": rpc error: code = NotFound desc = could not find container \"702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f\": container with ID starting with 702c3a963c597cbcbfe2b40c93e90d523be58228cd91a7d3b2b4dca77d13dc5f not found: ID does not exist"
Sep 08 16:38:00.721708 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.721690    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"]
Sep 08 16:38:00.725145 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.725127    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1fa1601c-7494874bb-4dfr2"]
Sep 08 16:38:00.738258 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.738230    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"]
Sep 08 16:38:00.741776 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:00.741759    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1fa1601c-59d54d9469-qlv5x"]
Sep 08 16:38:02.506816 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:02.506771    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196" path="/var/lib/kubelet/pods/1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196/volumes"
Sep 08 16:38:02.507320 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:02.507299    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d01fced-12ba-4d0d-984e-fda60e20a64e" path="/var/lib/kubelet/pods/7d01fced-12ba-4d0d-984e-fda60e20a64e/volumes"
Sep 08 16:38:47.286019 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:38:47.285987    2575 scope.go:117] "RemoveContainer" containerID="52333ae6ca5cef21326b3cb8c8882dda0f60c7b1e5f4a185239baaf372e3b3b4"
Sep 08 16:39:47.314671 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:39:47.314637    2575 scope.go:117] "RemoveContainer" containerID="b22fb195da90c43a5b3d0b7003d3967fcecd83b97f8ccf5808c38833e6e4c7ee"
Sep 08 16:39:47.331091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:39:47.331071    2575 scope.go:117] "RemoveContainer" containerID="b0192dfbdb2d53f1752ab2be0969d85f783c29f9fc74c1506f6d5181d4b3b601"
Sep 08 16:39:47.338666 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:39:47.338647    2575 scope.go:117] "RemoveContainer" containerID="55009650fd2bb9e3fadb7e37e7ce75c0553d9a50172769a15e75ab8931786db6"
Sep 08 16:40:06.352602 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.352542    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"]
Sep 08 16:40:06.353003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.352989    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196" containerName="payload-pre-processing"
Sep 08 16:40:06.353051 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.353004    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196" containerName="payload-pre-processing"
Sep 08 16:40:06.353051 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.353021    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5ca198a-bed4-45b6-9d5f-49f515abeb6b" containerName="revoke-keys"
Sep 08 16:40:06.353051 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.353026    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5ca198a-bed4-45b6-9d5f-49f515abeb6b" containerName="revoke-keys"
Sep 08 16:40:06.353051 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.353041    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d01fced-12ba-4d0d-984e-fda60e20a64e" containerName="payload-processing"
Sep 08 16:40:06.353051 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.353047    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d01fced-12ba-4d0d-984e-fda60e20a64e" containerName="payload-processing"
Sep 08 16:40:06.353291 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.353108    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5ca198a-bed4-45b6-9d5f-49f515abeb6b" containerName="revoke-keys"
Sep 08 16:40:06.353291 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.353115    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d01fced-12ba-4d0d-984e-fda60e20a64e" containerName="payload-processing"
Sep 08 16:40:06.353291 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.353121    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e5e8ce2-ae6a-4f19-bcae-2b8bf7daf196" containerName="payload-pre-processing"
Sep 08 16:40:06.356312 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.356292    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:06.359437 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.359420    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-402a97-dockercfg-qx9hw\""
Sep 08 16:40:06.359536 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.359517    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-402a97\""
Sep 08 16:40:06.366365 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.366339    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"]
Sep 08 16:40:06.370813 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.370792    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"]
Sep 08 16:40:06.370936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.370897    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:06.383976 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.383953    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"]
Sep 08 16:40:06.472197 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.472159    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4dqw\" (UniqueName: \"kubernetes.io/projected/eb914389-19fb-416e-aaf4-562bdbbd80f0-kube-api-access-l4dqw\") pod \"payload-processing-e2e-api-a-402a97-797bb469b5-6wnln\" (UID: \"eb914389-19fb-416e-aaf4-562bdbbd80f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:06.472397 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.472221    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbzfj\" (UniqueName: \"kubernetes.io/projected/3a41aa75-b67e-48f1-87c3-680164d53da9-kube-api-access-wbzfj\") pod \"payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696\" (UID: \"3a41aa75-b67e-48f1-87c3-680164d53da9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:06.472397 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.472322    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41aa75-b67e-48f1-87c3-680164d53da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696\" (UID: \"3a41aa75-b67e-48f1-87c3-680164d53da9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:06.472523 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.472423    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb914389-19fb-416e-aaf4-562bdbbd80f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-402a97-797bb469b5-6wnln\" (UID: \"eb914389-19fb-416e-aaf4-562bdbbd80f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:06.573925 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.573890    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb914389-19fb-416e-aaf4-562bdbbd80f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-402a97-797bb469b5-6wnln\" (UID: \"eb914389-19fb-416e-aaf4-562bdbbd80f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:06.574086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.573946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4dqw\" (UniqueName: \"kubernetes.io/projected/eb914389-19fb-416e-aaf4-562bdbbd80f0-kube-api-access-l4dqw\") pod \"payload-processing-e2e-api-a-402a97-797bb469b5-6wnln\" (UID: \"eb914389-19fb-416e-aaf4-562bdbbd80f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:06.574086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.573978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbzfj\" (UniqueName: \"kubernetes.io/projected/3a41aa75-b67e-48f1-87c3-680164d53da9-kube-api-access-wbzfj\") pod \"payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696\" (UID: \"3a41aa75-b67e-48f1-87c3-680164d53da9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:06.574086 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.574016    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41aa75-b67e-48f1-87c3-680164d53da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696\" (UID: \"3a41aa75-b67e-48f1-87c3-680164d53da9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:06.574678 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.574657    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb914389-19fb-416e-aaf4-562bdbbd80f0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-402a97-797bb469b5-6wnln\" (UID: \"eb914389-19fb-416e-aaf4-562bdbbd80f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:06.574767 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.574749    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41aa75-b67e-48f1-87c3-680164d53da9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696\" (UID: \"3a41aa75-b67e-48f1-87c3-680164d53da9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:06.581869 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.581848    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbzfj\" (UniqueName: \"kubernetes.io/projected/3a41aa75-b67e-48f1-87c3-680164d53da9-kube-api-access-wbzfj\") pod \"payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696\" (UID: \"3a41aa75-b67e-48f1-87c3-680164d53da9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:06.581975 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.581875    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4dqw\" (UniqueName: \"kubernetes.io/projected/eb914389-19fb-416e-aaf4-562bdbbd80f0-kube-api-access-l4dqw\") pod \"payload-processing-e2e-api-a-402a97-797bb469b5-6wnln\" (UID: \"eb914389-19fb-416e-aaf4-562bdbbd80f0\") " pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:06.667797 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.667722    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:06.684016 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.683942    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:06.955087 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:40:06.955060    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a41aa75_b67e_48f1_87c3_680164d53da9.slice/crio-5474ae15f3ad4709da3ec9548b4609d92d3c124516669530090aaadf18e1a75a WatchSource:0}: Error finding container 5474ae15f3ad4709da3ec9548b4609d92d3c124516669530090aaadf18e1a75a: Status 404 returned error can't find the container with id 5474ae15f3ad4709da3ec9548b4609d92d3c124516669530090aaadf18e1a75a
Sep 08 16:40:06.965811 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.965787    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"]
Sep 08 16:40:06.984622 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:06.984598    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"]
Sep 08 16:40:06.987959 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:40:06.987938    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb914389_19fb_416e_aaf4_562bdbbd80f0.slice/crio-cf62c7218ea2d5a0d1ded6f73f94ae3fdacce241df4aeac35288f9c579717bf6 WatchSource:0}: Error finding container cf62c7218ea2d5a0d1ded6f73f94ae3fdacce241df4aeac35288f9c579717bf6: Status 404 returned error can't find the container with id cf62c7218ea2d5a0d1ded6f73f94ae3fdacce241df4aeac35288f9c579717bf6
Sep 08 16:40:07.400864 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:07.400830    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln" event={"ID":"eb914389-19fb-416e-aaf4-562bdbbd80f0","Type":"ContainerStarted","Data":"bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6"}
Sep 08 16:40:07.401336 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:07.400880    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln" event={"ID":"eb914389-19fb-416e-aaf4-562bdbbd80f0","Type":"ContainerStarted","Data":"cf62c7218ea2d5a0d1ded6f73f94ae3fdacce241df4aeac35288f9c579717bf6"}
Sep 08 16:40:07.401336 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:07.401001    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:07.402431 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:07.402406    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696" event={"ID":"3a41aa75-b67e-48f1-87c3-680164d53da9","Type":"ContainerStarted","Data":"adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9"}
Sep 08 16:40:07.402498 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:07.402453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696" event={"ID":"3a41aa75-b67e-48f1-87c3-680164d53da9","Type":"ContainerStarted","Data":"5474ae15f3ad4709da3ec9548b4609d92d3c124516669530090aaadf18e1a75a"}
Sep 08 16:40:07.402498 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:07.402482    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:07.422540 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:07.422487    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln" podStartSLOduration=1.422469707 podStartE2EDuration="1.422469707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:07.421728683 +0000 UTC m=+1641.508392441" watchObservedRunningTime="2026-09-08 16:40:07.422469707 +0000 UTC m=+1641.509133450"
Sep 08 16:40:07.442229 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:07.442187    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696" podStartSLOduration=1.442173518 podStartE2EDuration="1.442173518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:40:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:40:07.440523185 +0000 UTC m=+1641.527186932" watchObservedRunningTime="2026-09-08 16:40:07.442173518 +0000 UTC m=+1641.528837270"
Sep 08 16:40:18.410151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:18.410118    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:40:18.410637 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:18.410282    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:40:47.367121 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:47.367089    2575 scope.go:117] "RemoveContainer" containerID="5f51fba72f8b11a18c7e675db2709dc2d241bbc47bf497843e040ca4ad1c8437"
Sep 08 16:40:47.375791 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:40:47.375773    2575 scope.go:117] "RemoveContainer" containerID="240f7801db62034afb5c6572dd75420dca30d380f38f68801d9dfe5800903890"
Sep 08 16:41:01.648953 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:01.648920    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"]
Sep 08 16:41:01.652678 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:01.652655    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"
Sep 08 16:41:01.655322 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:01.655302    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:41:01.661431 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:01.661407    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"]
Sep 08 16:41:01.744364 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:01.744325    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b790e76-69b4-4df4-931a-e06e7c21b7a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-402a97-tn9pj\" (UID: \"1b790e76-69b4-4df4-931a-e06e7c21b7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"
Sep 08 16:41:01.845810 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:01.845779    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b790e76-69b4-4df4-931a-e06e7c21b7a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-402a97-tn9pj\" (UID: \"1b790e76-69b4-4df4-931a-e06e7c21b7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"
Sep 08 16:41:01.846360 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:01.846340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b790e76-69b4-4df4-931a-e06e7c21b7a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-402a97-tn9pj\" (UID: \"1b790e76-69b4-4df4-931a-e06e7c21b7a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"
Sep 08 16:41:01.964070 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:01.963992    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"
Sep 08 16:41:02.239781 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:02.239715    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"]
Sep 08 16:41:02.242188 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:41:02.242161    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b790e76_69b4_4df4_931a_e06e7c21b7a0.slice/crio-e375b1a13e9c263f31020aa102e28d593e7aa76c9c8ba9fb94c31e4159176122 WatchSource:0}: Error finding container e375b1a13e9c263f31020aa102e28d593e7aa76c9c8ba9fb94c31e4159176122: Status 404 returned error can't find the container with id e375b1a13e9c263f31020aa102e28d593e7aa76c9c8ba9fb94c31e4159176122
Sep 08 16:41:02.707301 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:02.707264    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj" event={"ID":"1b790e76-69b4-4df4-931a-e06e7c21b7a0","Type":"ContainerStarted","Data":"a15190a268fc939e0577e1df231008a6b719b40f999485c0861b7b43a8ba942d"}
Sep 08 16:41:02.707703 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:02.707317    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj" event={"ID":"1b790e76-69b4-4df4-931a-e06e7c21b7a0","Type":"ContainerStarted","Data":"e375b1a13e9c263f31020aa102e28d593e7aa76c9c8ba9fb94c31e4159176122"}
Sep 08 16:41:02.725189 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:02.725133    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj" podStartSLOduration=1.725116839 podStartE2EDuration="1.725116839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:02.722466318 +0000 UTC m=+1696.809130066" watchObservedRunningTime="2026-09-08 16:41:02.725116839 +0000 UTC m=+1696.811780586"
Sep 08 16:41:03.721271 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:03.721232    2575 generic.go:356] "Generic (PLEG): container finished" podID="1b790e76-69b4-4df4-931a-e06e7c21b7a0" containerID="a15190a268fc939e0577e1df231008a6b719b40f999485c0861b7b43a8ba942d" exitCode=0
Sep 08 16:41:03.721719 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:03.721316    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj" event={"ID":"1b790e76-69b4-4df4-931a-e06e7c21b7a0","Type":"ContainerDied","Data":"a15190a268fc939e0577e1df231008a6b719b40f999485c0861b7b43a8ba942d"}
Sep 08 16:41:04.864102 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:04.864074    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"
Sep 08 16:41:04.970246 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:04.970218    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b790e76-69b4-4df4-931a-e06e7c21b7a0-maas-api-service-ca\") pod \"1b790e76-69b4-4df4-931a-e06e7c21b7a0\" (UID: \"1b790e76-69b4-4df4-931a-e06e7c21b7a0\") "
Sep 08 16:41:04.970540 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:04.970519    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b790e76-69b4-4df4-931a-e06e7c21b7a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b790e76-69b4-4df4-931a-e06e7c21b7a0" (UID: "1b790e76-69b4-4df4-931a-e06e7c21b7a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:05.071189 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:05.071124    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b790e76-69b4-4df4-931a-e06e7c21b7a0-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:05.732623 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:05.732565    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"
Sep 08 16:41:05.732801 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:05.732622    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj" event={"ID":"1b790e76-69b4-4df4-931a-e06e7c21b7a0","Type":"ContainerDied","Data":"e375b1a13e9c263f31020aa102e28d593e7aa76c9c8ba9fb94c31e4159176122"}
Sep 08 16:41:05.732801 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:05.732670    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e375b1a13e9c263f31020aa102e28d593e7aa76c9c8ba9fb94c31e4159176122"
Sep 08 16:41:06.373651 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.373621    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"]
Sep 08 16:41:06.374074 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.374060    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b790e76-69b4-4df4-931a-e06e7c21b7a0" containerName="revoke-keys"
Sep 08 16:41:06.374157 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.374075    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b790e76-69b4-4df4-931a-e06e7c21b7a0" containerName="revoke-keys"
Sep 08 16:41:06.374231 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.374221    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b790e76-69b4-4df4-931a-e06e7c21b7a0" containerName="revoke-keys"
Sep 08 16:41:06.377442 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.377421    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.380785 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.380751    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"default-dockercfg-kphbb\""
Sep 08 16:41:06.380785 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.380755    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"openshift-service-ca.crt\""
Sep 08 16:41:06.380947 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.380841    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"rate-test-model-e22de8-kserve-self-signed-certs\""
Sep 08 16:41:06.380947 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.380781    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-e22de8\"/\"kube-root-ca.crt\""
Sep 08 16:41:06.387977 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.387950    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"]
Sep 08 16:41:06.483928 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.483893    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372db45b-0183-441c-b358-229b60835f60-tls-certs\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.484106 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.483960    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-dshm\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.484106 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.483989    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj2bg\" (UniqueName: \"kubernetes.io/projected/372db45b-0183-441c-b358-229b60835f60-kube-api-access-zj2bg\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.484106 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.484006    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-home\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.484106 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.484048    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-tmp-dir\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.484106 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.484079    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-model-cache\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.584984 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.584950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj2bg\" (UniqueName: \"kubernetes.io/projected/372db45b-0183-441c-b358-229b60835f60-kube-api-access-zj2bg\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.584984 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.584988    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-home\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.585229 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.585125    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-tmp-dir\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.585229 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.585198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-model-cache\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.585347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.585291    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372db45b-0183-441c-b358-229b60835f60-tls-certs\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.585530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.585429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-dshm\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.585530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.585477    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-home\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.585530 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.585511    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-tmp-dir\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.585790 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.585624    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-model-cache\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.588138 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.588112    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-dshm\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.588458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.588438    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372db45b-0183-441c-b358-229b60835f60-tls-certs\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.598475 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.598446    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj2bg\" (UniqueName: \"kubernetes.io/projected/372db45b-0183-441c-b358-229b60835f60-kube-api-access-zj2bg\") pod \"rate-test-model-e22de8-kserve-647979cdc9-px59l\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") " pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.686717 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.686638    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:06.961191 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:06.961109    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"]
Sep 08 16:41:06.964249 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:41:06.964216    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod372db45b_0183_441c_b358_229b60835f60.slice/crio-7229194dcd789e45c6493291e215457c508b94add7312cca6adb26c08056decb WatchSource:0}: Error finding container 7229194dcd789e45c6493291e215457c508b94add7312cca6adb26c08056decb: Status 404 returned error can't find the container with id 7229194dcd789e45c6493291e215457c508b94add7312cca6adb26c08056decb
Sep 08 16:41:07.747247 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:07.747192    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l" event={"ID":"372db45b-0183-441c-b358-229b60835f60","Type":"ContainerStarted","Data":"6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b"}
Sep 08 16:41:07.747688 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:07.747261    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l" event={"ID":"372db45b-0183-441c-b358-229b60835f60","Type":"ContainerStarted","Data":"7229194dcd789e45c6493291e215457c508b94add7312cca6adb26c08056decb"}
Sep 08 16:41:07.773757 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:07.773702    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l" podStartSLOduration=1.773682288 podStartE2EDuration="1.773682288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:07.771877708 +0000 UTC m=+1701.858541455" watchObservedRunningTime="2026-09-08 16:41:07.773682288 +0000 UTC m=+1701.860346037"
Sep 08 16:41:16.687670 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:16.687636    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:16.688061 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:16.687681    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:16.700323 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:16.700295    2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:16.805417 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:16.805386    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:41:17.336087 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.336057    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"]
Sep 08 16:41:17.336386 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.336354    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln" podUID="eb914389-19fb-416e-aaf4-562bdbbd80f0" containerName="payload-processing" containerID="cri-o://bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6" gracePeriod=30
Sep 08 16:41:17.407393 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.407364    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"]
Sep 08 16:41:17.407645 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.407620    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696" podUID="3a41aa75-b67e-48f1-87c3-680164d53da9" containerName="payload-pre-processing" containerID="cri-o://adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9" gracePeriod=30
Sep 08 16:41:17.637017 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.636944    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"]
Sep 08 16:41:17.640571 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.640541    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-402a97-tn9pj"]
Sep 08 16:41:17.664940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.664920    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:41:17.709436 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.709412    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:41:17.786316 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.786289    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4dqw\" (UniqueName: \"kubernetes.io/projected/eb914389-19fb-416e-aaf4-562bdbbd80f0-kube-api-access-l4dqw\") pod \"eb914389-19fb-416e-aaf4-562bdbbd80f0\" (UID: \"eb914389-19fb-416e-aaf4-562bdbbd80f0\") "
Sep 08 16:41:17.786470 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.786395    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41aa75-b67e-48f1-87c3-680164d53da9-plugins-config-volume\") pod \"3a41aa75-b67e-48f1-87c3-680164d53da9\" (UID: \"3a41aa75-b67e-48f1-87c3-680164d53da9\") "
Sep 08 16:41:17.786470 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.786462    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb914389-19fb-416e-aaf4-562bdbbd80f0-plugins-config-volume\") pod \"eb914389-19fb-416e-aaf4-562bdbbd80f0\" (UID: \"eb914389-19fb-416e-aaf4-562bdbbd80f0\") "
Sep 08 16:41:17.786624 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.786525    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbzfj\" (UniqueName: \"kubernetes.io/projected/3a41aa75-b67e-48f1-87c3-680164d53da9-kube-api-access-wbzfj\") pod \"3a41aa75-b67e-48f1-87c3-680164d53da9\" (UID: \"3a41aa75-b67e-48f1-87c3-680164d53da9\") "
Sep 08 16:41:17.786865 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.786829    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a41aa75-b67e-48f1-87c3-680164d53da9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3a41aa75-b67e-48f1-87c3-680164d53da9" (UID: "3a41aa75-b67e-48f1-87c3-680164d53da9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:17.786930 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.786831    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb914389-19fb-416e-aaf4-562bdbbd80f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb914389-19fb-416e-aaf4-562bdbbd80f0" (UID: "eb914389-19fb-416e-aaf4-562bdbbd80f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:17.788328 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.788308    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb914389-19fb-416e-aaf4-562bdbbd80f0-kube-api-access-l4dqw" (OuterVolumeSpecName: "kube-api-access-l4dqw") pod "eb914389-19fb-416e-aaf4-562bdbbd80f0" (UID: "eb914389-19fb-416e-aaf4-562bdbbd80f0"). InnerVolumeSpecName "kube-api-access-l4dqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:41:17.788463 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.788441    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a41aa75-b67e-48f1-87c3-680164d53da9-kube-api-access-wbzfj" (OuterVolumeSpecName: "kube-api-access-wbzfj") pod "3a41aa75-b67e-48f1-87c3-680164d53da9" (UID: "3a41aa75-b67e-48f1-87c3-680164d53da9"). InnerVolumeSpecName "kube-api-access-wbzfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:41:17.800510 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.800484    2575 generic.go:356] "Generic (PLEG): container finished" podID="eb914389-19fb-416e-aaf4-562bdbbd80f0" containerID="bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6" exitCode=0
Sep 08 16:41:17.800642 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.800523    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"
Sep 08 16:41:17.800642 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.800537    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln" event={"ID":"eb914389-19fb-416e-aaf4-562bdbbd80f0","Type":"ContainerDied","Data":"bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6"}
Sep 08 16:41:17.800642 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.800603    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln" event={"ID":"eb914389-19fb-416e-aaf4-562bdbbd80f0","Type":"ContainerDied","Data":"cf62c7218ea2d5a0d1ded6f73f94ae3fdacce241df4aeac35288f9c579717bf6"}
Sep 08 16:41:17.800642 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.800638    2575 scope.go:117] "RemoveContainer" containerID="bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6"
Sep 08 16:41:17.802063 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.802041    2575 generic.go:356] "Generic (PLEG): container finished" podID="3a41aa75-b67e-48f1-87c3-680164d53da9" containerID="adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9" exitCode=0
Sep 08 16:41:17.802157 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.802074    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696" event={"ID":"3a41aa75-b67e-48f1-87c3-680164d53da9","Type":"ContainerDied","Data":"adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9"}
Sep 08 16:41:17.802157 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.802098    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"
Sep 08 16:41:17.802157 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.802123    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696" event={"ID":"3a41aa75-b67e-48f1-87c3-680164d53da9","Type":"ContainerDied","Data":"5474ae15f3ad4709da3ec9548b4609d92d3c124516669530090aaadf18e1a75a"}
Sep 08 16:41:17.809534 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.809518    2575 scope.go:117] "RemoveContainer" containerID="bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6"
Sep 08 16:41:17.809898 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:41:17.809879    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6\": container with ID starting with bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6 not found: ID does not exist" containerID="bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6"
Sep 08 16:41:17.809989 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.809904    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6"} err="failed to get container status \"bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6\": rpc error: code = NotFound desc = could not find container \"bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6\": container with ID starting with bc38d89a4f35e00ca2eb61a86f16bb7106f240764e3c540f91ebd239f53a0bd6 not found: ID does not exist"
Sep 08 16:41:17.809989 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.809920    2575 scope.go:117] "RemoveContainer" containerID="adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9"
Sep 08 16:41:17.819206 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.819162    2575 scope.go:117] "RemoveContainer" containerID="adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9"
Sep 08 16:41:17.819453 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:41:17.819426    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9\": container with ID starting with adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9 not found: ID does not exist" containerID="adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9"
Sep 08 16:41:17.819520 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.819459    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9"} err="failed to get container status \"adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9\": rpc error: code = NotFound desc = could not find container \"adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9\": container with ID starting with adc41835229b70294f585ec0d44e4177aa67ff0418693da786fb147c0ab0fbb9 not found: ID does not exist"
Sep 08 16:41:17.832414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.832393    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"]
Sep 08 16:41:17.837173 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.837148    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-402a97-849849b7d-6v696"]
Sep 08 16:41:17.851216 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.851194    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"]
Sep 08 16:41:17.854390 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.854369    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-402a97-797bb469b5-6wnln"]
Sep 08 16:41:17.887527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.887479    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a41aa75-b67e-48f1-87c3-680164d53da9-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:17.887527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.887497    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb914389-19fb-416e-aaf4-562bdbbd80f0-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:17.887527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.887507    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbzfj\" (UniqueName: \"kubernetes.io/projected/3a41aa75-b67e-48f1-87c3-680164d53da9-kube-api-access-wbzfj\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:17.887527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:17.887516    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4dqw\" (UniqueName: \"kubernetes.io/projected/eb914389-19fb-416e-aaf4-562bdbbd80f0-kube-api-access-l4dqw\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:18.507743 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:18.507668    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b790e76-69b4-4df4-931a-e06e7c21b7a0" path="/var/lib/kubelet/pods/1b790e76-69b4-4df4-931a-e06e7c21b7a0/volumes"
Sep 08 16:41:18.508176 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:18.508154    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a41aa75-b67e-48f1-87c3-680164d53da9" path="/var/lib/kubelet/pods/3a41aa75-b67e-48f1-87c3-680164d53da9/volumes"
Sep 08 16:41:18.508631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:18.508593    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb914389-19fb-416e-aaf4-562bdbbd80f0" path="/var/lib/kubelet/pods/eb914389-19fb-416e-aaf4-562bdbbd80f0/volumes"
Sep 08 16:41:24.755735 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.755704    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"]
Sep 08 16:41:24.756279 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.756262    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb914389-19fb-416e-aaf4-562bdbbd80f0" containerName="payload-processing"
Sep 08 16:41:24.756324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.756280    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb914389-19fb-416e-aaf4-562bdbbd80f0" containerName="payload-processing"
Sep 08 16:41:24.756324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.756292    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a41aa75-b67e-48f1-87c3-680164d53da9" containerName="payload-pre-processing"
Sep 08 16:41:24.756324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.756298    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a41aa75-b67e-48f1-87c3-680164d53da9" containerName="payload-pre-processing"
Sep 08 16:41:24.756417 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.756404    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a41aa75-b67e-48f1-87c3-680164d53da9" containerName="payload-pre-processing"
Sep 08 16:41:24.756455 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.756424    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb914389-19fb-416e-aaf4-562bdbbd80f0" containerName="payload-processing"
Sep 08 16:41:24.759604 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.759586    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"
Sep 08 16:41:24.762270 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.762250    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:41:24.767528 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.767504    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"]
Sep 08 16:41:24.851358 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.851332    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7117692c-444a-4b1e-8c4d-0337f60d1470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899\" (UID: \"7117692c-444a-4b1e-8c4d-0337f60d1470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"
Sep 08 16:41:24.952198 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.952168    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7117692c-444a-4b1e-8c4d-0337f60d1470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899\" (UID: \"7117692c-444a-4b1e-8c4d-0337f60d1470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"
Sep 08 16:41:24.952824 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:24.952798    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7117692c-444a-4b1e-8c4d-0337f60d1470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899\" (UID: \"7117692c-444a-4b1e-8c4d-0337f60d1470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"
Sep 08 16:41:25.071876 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:25.071812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"
Sep 08 16:41:25.343716 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:25.343692    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"]
Sep 08 16:41:25.345510 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:41:25.345480    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7117692c_444a_4b1e_8c4d_0337f60d1470.slice/crio-87855aa1c1296e0c19c696b974c326dccac12b65dae10307634f3117755080ab WatchSource:0}: Error finding container 87855aa1c1296e0c19c696b974c326dccac12b65dae10307634f3117755080ab: Status 404 returned error can't find the container with id 87855aa1c1296e0c19c696b974c326dccac12b65dae10307634f3117755080ab
Sep 08 16:41:25.847213 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:25.847178    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899" event={"ID":"7117692c-444a-4b1e-8c4d-0337f60d1470","Type":"ContainerStarted","Data":"0b08764330fab0a32df2c01da0d242de9bb008dcbdbcaf6c2b0c1cc428b513b8"}
Sep 08 16:41:25.847512 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:25.847238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899" event={"ID":"7117692c-444a-4b1e-8c4d-0337f60d1470","Type":"ContainerStarted","Data":"87855aa1c1296e0c19c696b974c326dccac12b65dae10307634f3117755080ab"}
Sep 08 16:41:25.862589 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:25.862529    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899" podStartSLOduration=1.862513527 podStartE2EDuration="1.862513527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:25.862036736 +0000 UTC m=+1719.948700488" watchObservedRunningTime="2026-09-08 16:41:25.862513527 +0000 UTC m=+1719.949177275"
Sep 08 16:41:26.853984 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:26.853949    2575 generic.go:356] "Generic (PLEG): container finished" podID="7117692c-444a-4b1e-8c4d-0337f60d1470" containerID="0b08764330fab0a32df2c01da0d242de9bb008dcbdbcaf6c2b0c1cc428b513b8" exitCode=0
Sep 08 16:41:26.854388 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:26.854029    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899" event={"ID":"7117692c-444a-4b1e-8c4d-0337f60d1470","Type":"ContainerDied","Data":"0b08764330fab0a32df2c01da0d242de9bb008dcbdbcaf6c2b0c1cc428b513b8"}
Sep 08 16:41:28.000409 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:28.000383    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"
Sep 08 16:41:28.178815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:28.178731    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7117692c-444a-4b1e-8c4d-0337f60d1470-maas-api-service-ca\") pod \"7117692c-444a-4b1e-8c4d-0337f60d1470\" (UID: \"7117692c-444a-4b1e-8c4d-0337f60d1470\") "
Sep 08 16:41:28.179052 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:28.179029    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7117692c-444a-4b1e-8c4d-0337f60d1470-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7117692c-444a-4b1e-8c4d-0337f60d1470" (UID: "7117692c-444a-4b1e-8c4d-0337f60d1470"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:28.279469 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:28.279442    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7117692c-444a-4b1e-8c4d-0337f60d1470-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:28.866043 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:28.866011    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"
Sep 08 16:41:28.866190 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:28.866061    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899" event={"ID":"7117692c-444a-4b1e-8c4d-0337f60d1470","Type":"ContainerDied","Data":"87855aa1c1296e0c19c696b974c326dccac12b65dae10307634f3117755080ab"}
Sep 08 16:41:28.866190 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:28.866115    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87855aa1c1296e0c19c696b974c326dccac12b65dae10307634f3117755080ab"
Sep 08 16:41:40.564223 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:40.564187    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"]
Sep 08 16:41:40.566235 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:40.566211    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7d9b1-tf899"]
Sep 08 16:41:42.507486 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:42.507451    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7117692c-444a-4b1e-8c4d-0337f60d1470" path="/var/lib/kubelet/pods/7117692c-444a-4b1e-8c4d-0337f60d1470/volumes"
Sep 08 16:41:47.719770 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.719691    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"]
Sep 08 16:41:47.720110 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.720085    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7117692c-444a-4b1e-8c4d-0337f60d1470" containerName="revoke-keys"
Sep 08 16:41:47.720110 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.720104    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7117692c-444a-4b1e-8c4d-0337f60d1470" containerName="revoke-keys"
Sep 08 16:41:47.720208 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.720197    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7117692c-444a-4b1e-8c4d-0337f60d1470" containerName="revoke-keys"
Sep 08 16:41:47.724885 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.724866    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"
Sep 08 16:41:47.727504 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.727484    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:41:47.731947 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.731925    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"]
Sep 08 16:41:47.849411 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.849380    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509325a-eb73-4cec-8ac3-0181bff852f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k\" (UID: \"a509325a-eb73-4cec-8ac3-0181bff852f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"
Sep 08 16:41:47.949902 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.949874    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509325a-eb73-4cec-8ac3-0181bff852f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k\" (UID: \"a509325a-eb73-4cec-8ac3-0181bff852f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"
Sep 08 16:41:47.950651 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:47.950631    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509325a-eb73-4cec-8ac3-0181bff852f9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k\" (UID: \"a509325a-eb73-4cec-8ac3-0181bff852f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"
Sep 08 16:41:48.037064 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:48.037007    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"
Sep 08 16:41:48.308779 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:48.308756    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"]
Sep 08 16:41:48.310489 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:41:48.310463    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda509325a_eb73_4cec_8ac3_0181bff852f9.slice/crio-e1b64cb7561edc73492b262d103db96750f356077b183804f7e648e4998f6f5b WatchSource:0}: Error finding container e1b64cb7561edc73492b262d103db96750f356077b183804f7e648e4998f6f5b: Status 404 returned error can't find the container with id e1b64cb7561edc73492b262d103db96750f356077b183804f7e648e4998f6f5b
Sep 08 16:41:48.975482 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:48.975401    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k" event={"ID":"a509325a-eb73-4cec-8ac3-0181bff852f9","Type":"ContainerStarted","Data":"cabf0664712a86bce9908182ae56d82149684a9bb2b35b97d12457b463f7bdcf"}
Sep 08 16:41:48.975482 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:48.975453    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k" event={"ID":"a509325a-eb73-4cec-8ac3-0181bff852f9","Type":"ContainerStarted","Data":"e1b64cb7561edc73492b262d103db96750f356077b183804f7e648e4998f6f5b"}
Sep 08 16:41:48.992297 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:48.992249    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k" podStartSLOduration=1.992235817 podStartE2EDuration="1.992235817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:41:48.990773162 +0000 UTC m=+1743.077436910" watchObservedRunningTime="2026-09-08 16:41:48.992235817 +0000 UTC m=+1743.078899565"
Sep 08 16:41:49.982395 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:49.982359    2575 generic.go:356] "Generic (PLEG): container finished" podID="a509325a-eb73-4cec-8ac3-0181bff852f9" containerID="cabf0664712a86bce9908182ae56d82149684a9bb2b35b97d12457b463f7bdcf" exitCode=0
Sep 08 16:41:49.982785 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:49.982439    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k" event={"ID":"a509325a-eb73-4cec-8ac3-0181bff852f9","Type":"ContainerDied","Data":"cabf0664712a86bce9908182ae56d82149684a9bb2b35b97d12457b463f7bdcf"}
Sep 08 16:41:51.129404 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:51.129377    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"
Sep 08 16:41:51.176424 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:51.176398    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509325a-eb73-4cec-8ac3-0181bff852f9-maas-api-service-ca\") pod \"a509325a-eb73-4cec-8ac3-0181bff852f9\" (UID: \"a509325a-eb73-4cec-8ac3-0181bff852f9\") "
Sep 08 16:41:51.176730 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:51.176704    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a509325a-eb73-4cec-8ac3-0181bff852f9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a509325a-eb73-4cec-8ac3-0181bff852f9" (UID: "a509325a-eb73-4cec-8ac3-0181bff852f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:51.176868 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:51.176846    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a509325a-eb73-4cec-8ac3-0181bff852f9-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:51.994766 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:51.994734    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"
Sep 08 16:41:51.994766 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:51.994749    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k" event={"ID":"a509325a-eb73-4cec-8ac3-0181bff852f9","Type":"ContainerDied","Data":"e1b64cb7561edc73492b262d103db96750f356077b183804f7e648e4998f6f5b"}
Sep 08 16:41:51.994986 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:51.994793    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1b64cb7561edc73492b262d103db96750f356077b183804f7e648e4998f6f5b"
Sep 08 16:41:53.160634 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:53.160601    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"]
Sep 08 16:41:58.283232 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.283197    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"]
Sep 08 16:41:58.283683 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.283404    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6" podUID="e9a36029-ccde-4a5f-8ce9-a909c4ed8e85" containerName="payload-processing" containerID="cri-o://2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b" gracePeriod=30
Sep 08 16:41:58.295964 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.295934    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"]
Sep 08 16:41:58.296232 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.296186    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh" podUID="a44117c9-b416-4f48-99bb-2f4a2c264102" containerName="payload-pre-processing" containerID="cri-o://6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8" gracePeriod=30
Sep 08 16:41:58.510883 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.510827    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"]
Sep 08 16:41:58.511075 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.510911    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8ac343-chz5k"]
Sep 08 16:41:58.581517 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.581494    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:41:58.604254 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.604227    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:41:58.643814 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.643777    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vft42\" (UniqueName: \"kubernetes.io/projected/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-kube-api-access-vft42\") pod \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\" (UID: \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\") "
Sep 08 16:41:58.643814 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.643807    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-plugins-config-volume\") pod \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\" (UID: \"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85\") "
Sep 08 16:41:58.644266 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.644237    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e9a36029-ccde-4a5f-8ce9-a909c4ed8e85" (UID: "e9a36029-ccde-4a5f-8ce9-a909c4ed8e85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:58.646013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.645986    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-kube-api-access-vft42" (OuterVolumeSpecName: "kube-api-access-vft42") pod "e9a36029-ccde-4a5f-8ce9-a909c4ed8e85" (UID: "e9a36029-ccde-4a5f-8ce9-a909c4ed8e85"). InnerVolumeSpecName "kube-api-access-vft42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:41:58.744831 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.744783    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kk895\" (UniqueName: \"kubernetes.io/projected/a44117c9-b416-4f48-99bb-2f4a2c264102-kube-api-access-kk895\") pod \"a44117c9-b416-4f48-99bb-2f4a2c264102\" (UID: \"a44117c9-b416-4f48-99bb-2f4a2c264102\") "
Sep 08 16:41:58.745004 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.744840    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44117c9-b416-4f48-99bb-2f4a2c264102-plugins-config-volume\") pod \"a44117c9-b416-4f48-99bb-2f4a2c264102\" (UID: \"a44117c9-b416-4f48-99bb-2f4a2c264102\") "
Sep 08 16:41:58.745126 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.745102    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vft42\" (UniqueName: \"kubernetes.io/projected/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-kube-api-access-vft42\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:58.745126 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.745125    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:58.745328 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.745299    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44117c9-b416-4f48-99bb-2f4a2c264102-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a44117c9-b416-4f48-99bb-2f4a2c264102" (UID: "a44117c9-b416-4f48-99bb-2f4a2c264102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:41:58.746967 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.746945    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44117c9-b416-4f48-99bb-2f4a2c264102-kube-api-access-kk895" (OuterVolumeSpecName: "kube-api-access-kk895") pod "a44117c9-b416-4f48-99bb-2f4a2c264102" (UID: "a44117c9-b416-4f48-99bb-2f4a2c264102"). InnerVolumeSpecName "kube-api-access-kk895". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:41:58.845880 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.845847    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk895\" (UniqueName: \"kubernetes.io/projected/a44117c9-b416-4f48-99bb-2f4a2c264102-kube-api-access-kk895\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:58.845880 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:58.845880    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44117c9-b416-4f48-99bb-2f4a2c264102-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:41:59.034151 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.034107    2575 generic.go:356] "Generic (PLEG): container finished" podID="e9a36029-ccde-4a5f-8ce9-a909c4ed8e85" containerID="2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b" exitCode=0
Sep 08 16:41:59.034337 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.034161    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"
Sep 08 16:41:59.034337 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.034190    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6" event={"ID":"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85","Type":"ContainerDied","Data":"2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b"}
Sep 08 16:41:59.034337 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.034283    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6" event={"ID":"e9a36029-ccde-4a5f-8ce9-a909c4ed8e85","Type":"ContainerDied","Data":"d431c0a31b94fa36fea8833e47d3b6b72c9478e1ff8ce8503b9bb7a7e71f85c1"}
Sep 08 16:41:59.035093 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.034309    2575 scope.go:117] "RemoveContainer" containerID="2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b"
Sep 08 16:41:59.035676 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.035652    2575 generic.go:356] "Generic (PLEG): container finished" podID="a44117c9-b416-4f48-99bb-2f4a2c264102" containerID="6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8" exitCode=0
Sep 08 16:41:59.035767 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.035711    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh" event={"ID":"a44117c9-b416-4f48-99bb-2f4a2c264102","Type":"ContainerDied","Data":"6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8"}
Sep 08 16:41:59.035767 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.035716    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"
Sep 08 16:41:59.035767 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.035749    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh" event={"ID":"a44117c9-b416-4f48-99bb-2f4a2c264102","Type":"ContainerDied","Data":"fa9f44710d6c98505a51d962c2a91c0418bf0fd3aeb071939380fe6491661a58"}
Sep 08 16:41:59.045827 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.045811    2575 scope.go:117] "RemoveContainer" containerID="2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b"
Sep 08 16:41:59.046143 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:41:59.046121    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b\": container with ID starting with 2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b not found: ID does not exist" containerID="2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b"
Sep 08 16:41:59.046211 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.046150    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b"} err="failed to get container status \"2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b\": rpc error: code = NotFound desc = could not find container \"2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b\": container with ID starting with 2d562c8a1c6c6785ea7b4ec9764e9095e5658b9467c5c8fc4d7cefc4b4f0b64b not found: ID does not exist"
Sep 08 16:41:59.046211 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.046169    2575 scope.go:117] "RemoveContainer" containerID="6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8"
Sep 08 16:41:59.054513 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.054496    2575 scope.go:117] "RemoveContainer" containerID="6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8"
Sep 08 16:41:59.054782 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:41:59.054762    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8\": container with ID starting with 6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8 not found: ID does not exist" containerID="6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8"
Sep 08 16:41:59.054849 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.054788    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8"} err="failed to get container status \"6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8\": rpc error: code = NotFound desc = could not find container \"6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8\": container with ID starting with 6299966a3f77786725632926a002a764f839e587dbb0cb5924263f8efbcf6ab8 not found: ID does not exist"
Sep 08 16:41:59.058242 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.058220    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"]
Sep 08 16:41:59.065789 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.065768    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-8ac343-748d67b69c-lqkv6"]
Sep 08 16:41:59.077008 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.076981    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"]
Sep 08 16:41:59.092286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.092263    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8ac343-79597ff846-pq4rh"]
Sep 08 16:41:59.440726 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.440692    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"]
Sep 08 16:41:59.441278 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441257    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a36029-ccde-4a5f-8ce9-a909c4ed8e85" containerName="payload-processing"
Sep 08 16:41:59.441361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441281    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a36029-ccde-4a5f-8ce9-a909c4ed8e85" containerName="payload-processing"
Sep 08 16:41:59.441361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441308    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a509325a-eb73-4cec-8ac3-0181bff852f9" containerName="revoke-keys"
Sep 08 16:41:59.441361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441317    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a509325a-eb73-4cec-8ac3-0181bff852f9" containerName="revoke-keys"
Sep 08 16:41:59.441361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441333    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a44117c9-b416-4f48-99bb-2f4a2c264102" containerName="payload-pre-processing"
Sep 08 16:41:59.441361 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441342    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a44117c9-b416-4f48-99bb-2f4a2c264102" containerName="payload-pre-processing"
Sep 08 16:41:59.441654 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441436    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a44117c9-b416-4f48-99bb-2f4a2c264102" containerName="payload-pre-processing"
Sep 08 16:41:59.441654 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441450    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a36029-ccde-4a5f-8ce9-a909c4ed8e85" containerName="payload-processing"
Sep 08 16:41:59.441654 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.441460    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a509325a-eb73-4cec-8ac3-0181bff852f9" containerName="revoke-keys"
Sep 08 16:41:59.446021 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.446001    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:41:59.448784 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.448766    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-352142-dockercfg-8zqv6\""
Sep 08 16:41:59.449124 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.449108    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-352142\""
Sep 08 16:41:59.455262 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.455241    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"]
Sep 08 16:41:59.551286 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.551182    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cb0dc4-2d67-4cef-bbcc-8105480ad028-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv\" (UID: \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:41:59.551463 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.551354    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q27x\" (UniqueName: \"kubernetes.io/projected/27cb0dc4-2d67-4cef-bbcc-8105480ad028-kube-api-access-8q27x\") pod \"payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv\" (UID: \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:41:59.652491 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.652456    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q27x\" (UniqueName: \"kubernetes.io/projected/27cb0dc4-2d67-4cef-bbcc-8105480ad028-kube-api-access-8q27x\") pod \"payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv\" (UID: \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:41:59.652725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.652524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cb0dc4-2d67-4cef-bbcc-8105480ad028-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv\" (UID: \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:41:59.653222 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.653202    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cb0dc4-2d67-4cef-bbcc-8105480ad028-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv\" (UID: \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:41:59.660944 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.660917    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q27x\" (UniqueName: \"kubernetes.io/projected/27cb0dc4-2d67-4cef-bbcc-8105480ad028-kube-api-access-8q27x\") pod \"payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv\" (UID: \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:41:59.757458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:41:59.757424    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:42:00.028355 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:00.028331    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"]
Sep 08 16:42:00.031512 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:42:00.031477    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cb0dc4_2d67_4cef_bbcc_8105480ad028.slice/crio-85f5266430e97b57f317ac715c1796cd5ef3506d728f9a4349e1f92c77220c0a WatchSource:0}: Error finding container 85f5266430e97b57f317ac715c1796cd5ef3506d728f9a4349e1f92c77220c0a: Status 404 returned error can't find the container with id 85f5266430e97b57f317ac715c1796cd5ef3506d728f9a4349e1f92c77220c0a
Sep 08 16:42:00.057675 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:00.057612    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv" event={"ID":"27cb0dc4-2d67-4cef-bbcc-8105480ad028","Type":"ContainerStarted","Data":"85f5266430e97b57f317ac715c1796cd5ef3506d728f9a4349e1f92c77220c0a"}
Sep 08 16:42:00.507515 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:00.507483    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a44117c9-b416-4f48-99bb-2f4a2c264102" path="/var/lib/kubelet/pods/a44117c9-b416-4f48-99bb-2f4a2c264102/volumes"
Sep 08 16:42:00.507884 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:00.507826    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a509325a-eb73-4cec-8ac3-0181bff852f9" path="/var/lib/kubelet/pods/a509325a-eb73-4cec-8ac3-0181bff852f9/volumes"
Sep 08 16:42:00.508152 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:00.508139    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a36029-ccde-4a5f-8ce9-a909c4ed8e85" path="/var/lib/kubelet/pods/e9a36029-ccde-4a5f-8ce9-a909c4ed8e85/volumes"
Sep 08 16:42:01.064846 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:01.064810    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv" event={"ID":"27cb0dc4-2d67-4cef-bbcc-8105480ad028","Type":"ContainerStarted","Data":"f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641"}
Sep 08 16:42:01.065006 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:01.064869    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:42:01.089183 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:01.089136    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv" podStartSLOduration=2.089115912 podStartE2EDuration="2.089115912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:01.087649636 +0000 UTC m=+1755.174313385" watchObservedRunningTime="2026-09-08 16:42:01.089115912 +0000 UTC m=+1755.175779660"
Sep 08 16:42:05.603150 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:05.603111    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"]
Sep 08 16:42:05.607839 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:05.607822    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"
Sep 08 16:42:05.610336 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:05.610315    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:42:05.614567 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:05.614541    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"]
Sep 08 16:42:05.712526 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:05.712482    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9\" (UID: \"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"
Sep 08 16:42:05.813428 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:05.813393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9\" (UID: \"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"
Sep 08 16:42:05.814027 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:05.814007    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9\" (UID: \"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"
Sep 08 16:42:05.921990 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:05.921914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"
Sep 08 16:42:06.198466 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:06.198293    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"]
Sep 08 16:42:06.201337 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:42:06.201297    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d360cd_b4ad_4d64_b91f_f3512c4b2a8e.slice/crio-6dd6283d4d12a30199c2bb050bc0f086f6b70cf390baf17e5926cd75c9a9707f WatchSource:0}: Error finding container 6dd6283d4d12a30199c2bb050bc0f086f6b70cf390baf17e5926cd75c9a9707f: Status 404 returned error can't find the container with id 6dd6283d4d12a30199c2bb050bc0f086f6b70cf390baf17e5926cd75c9a9707f
Sep 08 16:42:07.100050 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:07.100015    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9" event={"ID":"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e","Type":"ContainerStarted","Data":"505b3cb81e3349c8f4a2db2422c15990f2d254035f0d49e4be9288aabdfa98e3"}
Sep 08 16:42:07.100411 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:07.100077    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9" event={"ID":"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e","Type":"ContainerStarted","Data":"6dd6283d4d12a30199c2bb050bc0f086f6b70cf390baf17e5926cd75c9a9707f"}
Sep 08 16:42:07.115406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:07.115362    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9" podStartSLOduration=2.11534814 podStartE2EDuration="2.11534814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:42:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:07.11489141 +0000 UTC m=+1761.201555158" watchObservedRunningTime="2026-09-08 16:42:07.11534814 +0000 UTC m=+1761.202011887"
Sep 08 16:42:08.106952 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.106920    2575 generic.go:356] "Generic (PLEG): container finished" podID="d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e" containerID="505b3cb81e3349c8f4a2db2422c15990f2d254035f0d49e4be9288aabdfa98e3" exitCode=0
Sep 08 16:42:08.107321 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.107015    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9" event={"ID":"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e","Type":"ContainerDied","Data":"505b3cb81e3349c8f4a2db2422c15990f2d254035f0d49e4be9288aabdfa98e3"}
Sep 08 16:42:08.179951 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.179913    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l" podUID="372db45b-0183-441c-b358-229b60835f60" containerName="main" containerID="cri-o://6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b" gracePeriod=45
Sep 08 16:42:08.443057 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.443028    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:42:08.538761 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.538731    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-dshm\") pod \"372db45b-0183-441c-b358-229b60835f60\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") "
Sep 08 16:42:08.538927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.538769    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372db45b-0183-441c-b358-229b60835f60-tls-certs\") pod \"372db45b-0183-441c-b358-229b60835f60\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") "
Sep 08 16:42:08.538927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.538798    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-model-cache\") pod \"372db45b-0183-441c-b358-229b60835f60\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") "
Sep 08 16:42:08.538927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.538853    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-tmp-dir\") pod \"372db45b-0183-441c-b358-229b60835f60\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") "
Sep 08 16:42:08.538927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.538872    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zj2bg\" (UniqueName: \"kubernetes.io/projected/372db45b-0183-441c-b358-229b60835f60-kube-api-access-zj2bg\") pod \"372db45b-0183-441c-b358-229b60835f60\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") "
Sep 08 16:42:08.538927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.538911    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-home\") pod \"372db45b-0183-441c-b358-229b60835f60\" (UID: \"372db45b-0183-441c-b358-229b60835f60\") "
Sep 08 16:42:08.539200 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.539146    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-model-cache" (OuterVolumeSpecName: "model-cache") pod "372db45b-0183-441c-b358-229b60835f60" (UID: "372db45b-0183-441c-b358-229b60835f60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:08.539244 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.539172    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "372db45b-0183-441c-b358-229b60835f60" (UID: "372db45b-0183-441c-b358-229b60835f60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:08.539244 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.539218    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-home" (OuterVolumeSpecName: "home") pod "372db45b-0183-441c-b358-229b60835f60" (UID: "372db45b-0183-441c-b358-229b60835f60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:08.541023 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.540994    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372db45b-0183-441c-b358-229b60835f60-kube-api-access-zj2bg" (OuterVolumeSpecName: "kube-api-access-zj2bg") pod "372db45b-0183-441c-b358-229b60835f60" (UID: "372db45b-0183-441c-b358-229b60835f60"). InnerVolumeSpecName "kube-api-access-zj2bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:42:08.541023 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.541008    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372db45b-0183-441c-b358-229b60835f60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "372db45b-0183-441c-b358-229b60835f60" (UID: "372db45b-0183-441c-b358-229b60835f60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:42:08.541174 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.541069    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-dshm" (OuterVolumeSpecName: "dshm") pod "372db45b-0183-441c-b358-229b60835f60" (UID: "372db45b-0183-441c-b358-229b60835f60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:42:08.639719 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.639689    2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-tmp-dir\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:08.639719 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.639713    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj2bg\" (UniqueName: \"kubernetes.io/projected/372db45b-0183-441c-b358-229b60835f60-kube-api-access-zj2bg\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:08.639719 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.639723    2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-home\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:08.639910 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.639732    2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-dshm\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:08.639910 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.639739    2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/372db45b-0183-441c-b358-229b60835f60-tls-certs\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:08.639910 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:08.639747    2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/372db45b-0183-441c-b358-229b60835f60-model-cache\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:09.113341 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.113310    2575 generic.go:356] "Generic (PLEG): container finished" podID="372db45b-0183-441c-b358-229b60835f60" containerID="6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b" exitCode=0
Sep 08 16:42:09.113341 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.113351    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"
Sep 08 16:42:09.113874 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.113391    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l" event={"ID":"372db45b-0183-441c-b358-229b60835f60","Type":"ContainerDied","Data":"6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b"}
Sep 08 16:42:09.113874 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.113442    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l" event={"ID":"372db45b-0183-441c-b358-229b60835f60","Type":"ContainerDied","Data":"7229194dcd789e45c6493291e215457c508b94add7312cca6adb26c08056decb"}
Sep 08 16:42:09.113874 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.113465    2575 scope.go:117] "RemoveContainer" containerID="6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b"
Sep 08 16:42:09.124602 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.124560    2575 scope.go:117] "RemoveContainer" containerID="6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b"
Sep 08 16:42:09.124930 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:42:09.124899    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b\": container with ID starting with 6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b not found: ID does not exist" containerID="6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b"
Sep 08 16:42:09.125046 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.124941    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b"} err="failed to get container status \"6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b\": rpc error: code = NotFound desc = could not find container \"6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b\": container with ID starting with 6075ee4e152fab241833f032204ff53bdd3c9212c77c551c226f25b623f4ee7b not found: ID does not exist"
Sep 08 16:42:09.138836 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.138807    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"]
Sep 08 16:42:09.146020 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.145994    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-e22de8/rate-test-model-e22de8-kserve-647979cdc9-px59l"]
Sep 08 16:42:09.259566 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.259544    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"
Sep 08 16:42:09.346611 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.346562    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e-maas-api-service-ca\") pod \"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e\" (UID: \"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e\") "
Sep 08 16:42:09.346897 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.346875    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e" (UID: "d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:42:09.447690 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:09.447614    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:10.120439 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:10.120406    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"
Sep 08 16:42:10.120439 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:10.120429    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9" event={"ID":"d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e","Type":"ContainerDied","Data":"6dd6283d4d12a30199c2bb050bc0f086f6b70cf390baf17e5926cd75c9a9707f"}
Sep 08 16:42:10.120994 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:10.120475    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6dd6283d4d12a30199c2bb050bc0f086f6b70cf390baf17e5926cd75c9a9707f"
Sep 08 16:42:10.506713 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:10.506636    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="372db45b-0183-441c-b358-229b60835f60" path="/var/lib/kubelet/pods/372db45b-0183-441c-b358-229b60835f60/volumes"
Sep 08 16:42:12.074243 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:12.074218    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:42:16.177212 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.177182    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"]
Sep 08 16:42:16.177720 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.177409    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h" podUID="3af5520a-1ebe-44b3-82a1-90ef502b1dfe" containerName="payload-processing" containerID="cri-o://410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e" gracePeriod=30
Sep 08 16:42:16.184620 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.184574    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"]
Sep 08 16:42:16.184767 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.184753    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw" podUID="74cfa857-b9c7-430e-8644-ddd409fd7fe0" containerName="payload-pre-processing" containerID="cri-o://50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87" gracePeriod=30
Sep 08 16:42:16.365496 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.365465    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"]
Sep 08 16:42:16.367317 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.367296    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8ded33-n5kr9"]
Sep 08 16:42:16.464663 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.464641    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:42:16.467715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.467693    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:42:16.525103 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.524470    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e" path="/var/lib/kubelet/pods/d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e/volumes"
Sep 08 16:42:16.609401 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.609373    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-plugins-config-volume\") pod \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\" (UID: \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\") "
Sep 08 16:42:16.609537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.609452    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7c2c\" (UniqueName: \"kubernetes.io/projected/74cfa857-b9c7-430e-8644-ddd409fd7fe0-kube-api-access-d7c2c\") pod \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\" (UID: \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\") "
Sep 08 16:42:16.609537 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.609479    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cfa857-b9c7-430e-8644-ddd409fd7fe0-plugins-config-volume\") pod \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\" (UID: \"74cfa857-b9c7-430e-8644-ddd409fd7fe0\") "
Sep 08 16:42:16.609669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.609565    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tshw\" (UniqueName: \"kubernetes.io/projected/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-kube-api-access-5tshw\") pod \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\" (UID: \"3af5520a-1ebe-44b3-82a1-90ef502b1dfe\") "
Sep 08 16:42:16.609786 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.609766    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3af5520a-1ebe-44b3-82a1-90ef502b1dfe" (UID: "3af5520a-1ebe-44b3-82a1-90ef502b1dfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:42:16.609884 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.609871    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:16.609949 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.609925    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cfa857-b9c7-430e-8644-ddd409fd7fe0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "74cfa857-b9c7-430e-8644-ddd409fd7fe0" (UID: "74cfa857-b9c7-430e-8644-ddd409fd7fe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:42:16.611452 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.611423    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-kube-api-access-5tshw" (OuterVolumeSpecName: "kube-api-access-5tshw") pod "3af5520a-1ebe-44b3-82a1-90ef502b1dfe" (UID: "3af5520a-1ebe-44b3-82a1-90ef502b1dfe"). InnerVolumeSpecName "kube-api-access-5tshw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:42:16.611552 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.611471    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cfa857-b9c7-430e-8644-ddd409fd7fe0-kube-api-access-d7c2c" (OuterVolumeSpecName: "kube-api-access-d7c2c") pod "74cfa857-b9c7-430e-8644-ddd409fd7fe0" (UID: "74cfa857-b9c7-430e-8644-ddd409fd7fe0"). InnerVolumeSpecName "kube-api-access-d7c2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:42:16.710677 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.710616    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7c2c\" (UniqueName: \"kubernetes.io/projected/74cfa857-b9c7-430e-8644-ddd409fd7fe0-kube-api-access-d7c2c\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:16.710769 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.710691    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74cfa857-b9c7-430e-8644-ddd409fd7fe0-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:16.710769 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:16.710717    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tshw\" (UniqueName: \"kubernetes.io/projected/3af5520a-1ebe-44b3-82a1-90ef502b1dfe-kube-api-access-5tshw\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:42:17.168348 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.168313    2575 generic.go:356] "Generic (PLEG): container finished" podID="3af5520a-1ebe-44b3-82a1-90ef502b1dfe" containerID="410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e" exitCode=0
Sep 08 16:42:17.168518 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.168353    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"
Sep 08 16:42:17.168518 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.168405    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h" event={"ID":"3af5520a-1ebe-44b3-82a1-90ef502b1dfe","Type":"ContainerDied","Data":"410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e"}
Sep 08 16:42:17.168518 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.168458    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h" event={"ID":"3af5520a-1ebe-44b3-82a1-90ef502b1dfe","Type":"ContainerDied","Data":"229c62bb1cebadcd527c8c57bcfb7cb8e6c2dd20ce06fe63688c86d539cf8e7d"}
Sep 08 16:42:17.168518 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.168487    2575 scope.go:117] "RemoveContainer" containerID="410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e"
Sep 08 16:42:17.169822 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.169799    2575 generic.go:356] "Generic (PLEG): container finished" podID="74cfa857-b9c7-430e-8644-ddd409fd7fe0" containerID="50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87" exitCode=0
Sep 08 16:42:17.169937 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.169842    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"
Sep 08 16:42:17.169937 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.169852    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw" event={"ID":"74cfa857-b9c7-430e-8644-ddd409fd7fe0","Type":"ContainerDied","Data":"50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87"}
Sep 08 16:42:17.169937 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.169882    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw" event={"ID":"74cfa857-b9c7-430e-8644-ddd409fd7fe0","Type":"ContainerDied","Data":"64425c8d13e0ff51e90ca393c3d17c1d9be68409e5693d634cb5e6e2e9325495"}
Sep 08 16:42:17.177756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.177740    2575 scope.go:117] "RemoveContainer" containerID="410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e"
Sep 08 16:42:17.178071 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:42:17.178026    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e\": container with ID starting with 410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e not found: ID does not exist" containerID="410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e"
Sep 08 16:42:17.178330 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.178147    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e"} err="failed to get container status \"410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e\": rpc error: code = NotFound desc = could not find container \"410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e\": container with ID starting with 410c4542ef774b7749b4ebb25f8dec50664322de11cc71605f765951cbc9ce5e not found: ID does not exist"
Sep 08 16:42:17.178330 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.178177    2575 scope.go:117] "RemoveContainer" containerID="50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87"
Sep 08 16:42:17.185912 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.185896    2575 scope.go:117] "RemoveContainer" containerID="50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87"
Sep 08 16:42:17.186175 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:42:17.186156    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87\": container with ID starting with 50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87 not found: ID does not exist" containerID="50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87"
Sep 08 16:42:17.186222 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.186181    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87"} err="failed to get container status \"50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87\": rpc error: code = NotFound desc = could not find container \"50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87\": container with ID starting with 50291b4cbee77dd2f2b04d19bd1af7268b4bbf343e7708fb306e2aa077272b87 not found: ID does not exist"
Sep 08 16:42:17.235841 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.235816    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"]
Sep 08 16:42:17.254187 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.254156    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-8ded33-7bb49f4db9-qzj5h"]
Sep 08 16:42:17.298171 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.298147    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"]
Sep 08 16:42:17.303309 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:17.303285    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-8ded33-57b4cc9cf4-f52nw"]
Sep 08 16:42:18.506667 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:18.506636    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3af5520a-1ebe-44b3-82a1-90ef502b1dfe" path="/var/lib/kubelet/pods/3af5520a-1ebe-44b3-82a1-90ef502b1dfe/volumes"
Sep 08 16:42:18.510599 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:18.507508    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74cfa857-b9c7-430e-8644-ddd409fd7fe0" path="/var/lib/kubelet/pods/74cfa857-b9c7-430e-8644-ddd409fd7fe0/volumes"
Sep 08 16:42:27.100867 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.100834    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"]
Sep 08 16:42:27.101445 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101423    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3af5520a-1ebe-44b3-82a1-90ef502b1dfe" containerName="payload-processing"
Sep 08 16:42:27.101541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101447    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3af5520a-1ebe-44b3-82a1-90ef502b1dfe" containerName="payload-processing"
Sep 08 16:42:27.101541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101462    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="372db45b-0183-441c-b358-229b60835f60" containerName="main"
Sep 08 16:42:27.101541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101470    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="372db45b-0183-441c-b358-229b60835f60" containerName="main"
Sep 08 16:42:27.101541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101488    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e" containerName="revoke-keys"
Sep 08 16:42:27.101541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101496    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e" containerName="revoke-keys"
Sep 08 16:42:27.101541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101511    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74cfa857-b9c7-430e-8644-ddd409fd7fe0" containerName="payload-pre-processing"
Sep 08 16:42:27.101541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101519    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74cfa857-b9c7-430e-8644-ddd409fd7fe0" containerName="payload-pre-processing"
Sep 08 16:42:27.101929 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101655    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3af5520a-1ebe-44b3-82a1-90ef502b1dfe" containerName="payload-processing"
Sep 08 16:42:27.101929 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101669    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="372db45b-0183-441c-b358-229b60835f60" containerName="main"
Sep 08 16:42:27.101929 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101679    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4d360cd-b4ad-4d64-b91f-f3512c4b2a8e" containerName="revoke-keys"
Sep 08 16:42:27.101929 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.101693    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="74cfa857-b9c7-430e-8644-ddd409fd7fe0" containerName="payload-pre-processing"
Sep 08 16:42:27.105298 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.105276    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.108680 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.108661    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 16:42:27.108789 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.108665    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksssl\""
Sep 08 16:42:27.108789 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.108665    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-995b22-serving-cert\""
Sep 08 16:42:27.116078 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.115883    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"]
Sep 08 16:42:27.190526 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.190500    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vvwq\" (UniqueName: \"kubernetes.io/projected/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-kube-api-access-9vvwq\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.190686 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.190567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.190686 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.190662    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.190686 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.190678    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-metrics-certs\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.291555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.291524    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.291720 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.291620    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.291720 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.291646    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-metrics-certs\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.291720 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:42:27.291680    2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-serving-cert: secret "maas-api-e2e-ipp-b-995b22-serving-cert" not found
Sep 08 16:42:27.291892 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.291729    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vvwq\" (UniqueName: \"kubernetes.io/projected/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-kube-api-access-9vvwq\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.291892 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:42:27.291741    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls podName:784aac9b-2f12-4ce5-91c7-0fba9e40bcc3 nodeName:}" failed. No retries permitted until 2026-09-08 16:42:27.791726 +0000 UTC m=+1781.878389739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls") pod "maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s" (UID: "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3") : secret "maas-api-e2e-ipp-b-995b22-serving-cert" not found
Sep 08 16:42:27.292368 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.292340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.294067 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.294042    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-metrics-certs\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.302536 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.302516    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vvwq\" (UniqueName: \"kubernetes.io/projected/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-kube-api-access-9vvwq\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.774674 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.774638    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"]
Sep 08 16:42:27.778345 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.778324    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:27.782022 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.782004    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-995b22\""
Sep 08 16:42:27.789960 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.789937    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"]
Sep 08 16:42:27.796808 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.796780    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed129475-b937-44a7-8476-0c4cdf51366d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2\" (UID: \"ed129475-b937-44a7-8476-0c4cdf51366d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:27.796940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.796825    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.796940 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.796862    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/ed129475-b937-44a7-8476-0c4cdf51366d-kube-api-access-s6nhr\") pod \"payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2\" (UID: \"ed129475-b937-44a7-8476-0c4cdf51366d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:27.799056 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.799029    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls\") pod \"maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:27.818875 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.818853    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"]
Sep 08 16:42:27.822395 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.822377    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:27.826807 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.826784    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-995b22-dockercfg-k4bt4\""
Sep 08 16:42:27.842519 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.842494    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"]
Sep 08 16:42:27.897807 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.897777    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4ms\" (UniqueName: \"kubernetes.io/projected/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-kube-api-access-st4ms\") pod \"payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r\" (UID: \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:27.897949 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.897815    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/ed129475-b937-44a7-8476-0c4cdf51366d-kube-api-access-s6nhr\") pod \"payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2\" (UID: \"ed129475-b937-44a7-8476-0c4cdf51366d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:27.897949 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.897851    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r\" (UID: \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:27.897949 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.897919    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed129475-b937-44a7-8476-0c4cdf51366d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2\" (UID: \"ed129475-b937-44a7-8476-0c4cdf51366d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:27.898621 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.898603    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed129475-b937-44a7-8476-0c4cdf51366d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2\" (UID: \"ed129475-b937-44a7-8476-0c4cdf51366d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:27.907247 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.907225    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/ed129475-b937-44a7-8476-0c4cdf51366d-kube-api-access-s6nhr\") pod \"payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2\" (UID: \"ed129475-b937-44a7-8476-0c4cdf51366d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:27.998454 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.998429    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4ms\" (UniqueName: \"kubernetes.io/projected/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-kube-api-access-st4ms\") pod \"payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r\" (UID: \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:27.998546 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.998472    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r\" (UID: \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:27.999123 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:27.999105    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r\" (UID: \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:28.010663 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:28.010644    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4ms\" (UniqueName: \"kubernetes.io/projected/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-kube-api-access-st4ms\") pod \"payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r\" (UID: \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:28.022515 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:28.022491    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:28.089904 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:28.089860    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:28.132741 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:28.132711    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:28.295856 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:28.295833    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"]
Sep 08 16:42:28.297999 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:42:28.297965    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod784aac9b_2f12_4ce5_91c7_0fba9e40bcc3.slice/crio-4f3e41bbfd4b99693919998a55aca79161cb61d1967ef24b3942872e49a3c895 WatchSource:0}: Error finding container 4f3e41bbfd4b99693919998a55aca79161cb61d1967ef24b3942872e49a3c895: Status 404 returned error can't find the container with id 4f3e41bbfd4b99693919998a55aca79161cb61d1967ef24b3942872e49a3c895
Sep 08 16:42:28.299414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:28.299396    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:42:28.364859 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:28.364835    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"]
Sep 08 16:42:28.366611 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:42:28.366574    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded129475_b937_44a7_8476_0c4cdf51366d.slice/crio-e0c16a11f84e79584367bcbe5450ea8292e6e1f8516aa7e50167bd89ec09b5d0 WatchSource:0}: Error finding container e0c16a11f84e79584367bcbe5450ea8292e6e1f8516aa7e50167bd89ec09b5d0: Status 404 returned error can't find the container with id e0c16a11f84e79584367bcbe5450ea8292e6e1f8516aa7e50167bd89ec09b5d0
Sep 08 16:42:28.412412 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:28.412388    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"]
Sep 08 16:42:28.413019 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:42:28.412994    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb96ee5_b26d_4983_82a6_9cd05241ac2d.slice/crio-57275270f30d17e62dcf81b3639ebdef29c26a5d628ee28dd866e5f54a4ef44b WatchSource:0}: Error finding container 57275270f30d17e62dcf81b3639ebdef29c26a5d628ee28dd866e5f54a4ef44b: Status 404 returned error can't find the container with id 57275270f30d17e62dcf81b3639ebdef29c26a5d628ee28dd866e5f54a4ef44b
Sep 08 16:42:29.239376 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.239216    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r" event={"ID":"6cb96ee5-b26d-4983-82a6-9cd05241ac2d","Type":"ContainerStarted","Data":"492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5"}
Sep 08 16:42:29.239376 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.239315    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r" event={"ID":"6cb96ee5-b26d-4983-82a6-9cd05241ac2d","Type":"ContainerStarted","Data":"57275270f30d17e62dcf81b3639ebdef29c26a5d628ee28dd866e5f54a4ef44b"}
Sep 08 16:42:29.239376 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.239344    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:29.240845 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.240815    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2" event={"ID":"ed129475-b937-44a7-8476-0c4cdf51366d","Type":"ContainerStarted","Data":"eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4"}
Sep 08 16:42:29.240962 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.240863    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:29.240962 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.240886    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2" event={"ID":"ed129475-b937-44a7-8476-0c4cdf51366d","Type":"ContainerStarted","Data":"e0c16a11f84e79584367bcbe5450ea8292e6e1f8516aa7e50167bd89ec09b5d0"}
Sep 08 16:42:29.242042 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.242021    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s" event={"ID":"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3","Type":"ContainerStarted","Data":"4f3e41bbfd4b99693919998a55aca79161cb61d1967ef24b3942872e49a3c895"}
Sep 08 16:42:29.262076 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.262031    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r" podStartSLOduration=2.262014925 podStartE2EDuration="2.262014925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:29.259976173 +0000 UTC m=+1783.346639933" watchObservedRunningTime="2026-09-08 16:42:29.262014925 +0000 UTC m=+1783.348678675"
Sep 08 16:42:29.282110 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:29.282068    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2" podStartSLOduration=2.282055705 podStartE2EDuration="2.282055705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:29.28029102 +0000 UTC m=+1783.366954772" watchObservedRunningTime="2026-09-08 16:42:29.282055705 +0000 UTC m=+1783.368719453"
Sep 08 16:42:31.258744 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:31.258707    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s" event={"ID":"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3","Type":"ContainerStarted","Data":"30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39"}
Sep 08 16:42:31.259206 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:31.258795    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:31.278061 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:31.278006    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s" podStartSLOduration=2.19491004 podStartE2EDuration="4.277993082s" totalImagesPullingTime="2.083083042s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:42:31.27621861 +0000 UTC m=+1785.362882362" watchObservedRunningTime="2026-09-08 16:42:31.277993082 +0000 UTC m=+1785.364656883"
Sep 08 16:42:37.269900 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:37.269869    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:42:40.253277 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:40.253192    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:42:40.253668 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:40.253415    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:42:46.568963 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:46.568927    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:42:46.573900 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:46.573875    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:42:46.575525 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:46.575504    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:42:46.579746 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:46.579727    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:42:55.829095 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:55.829058    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xsx6m_71def932-1f2f-44ac-a578-0e90c4bfdf63/payload-processing/0.log"
Sep 08 16:42:55.944042 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:42:55.944006    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r_6cb96ee5-b26d-4983-82a6-9cd05241ac2d/payload-processing/0.log"
Sep 08 16:43:31.601308 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:31.601271    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xsx6m_71def932-1f2f-44ac-a578-0e90c4bfdf63/payload-processing/0.log"
Sep 08 16:43:31.718231 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:31.718208    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r_6cb96ee5-b26d-4983-82a6-9cd05241ac2d/payload-processing/0.log"
Sep 08 16:43:36.101012 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.100978    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"]
Sep 08 16:43:36.104801 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.104778    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.108351 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.108329    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-8dca27-serving-cert\""
Sep 08 16:43:36.115150 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.115126    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"]
Sep 08 16:43:36.190070 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.190037    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.190212 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.190117    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddb5009-d762-4d14-836f-d744209b4171-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.190212 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.190161    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.190212 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.190183    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crpkn\" (UniqueName: \"kubernetes.io/projected/6ddb5009-d762-4d14-836f-d744209b4171-kube-api-access-crpkn\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.291107 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.291072    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddb5009-d762-4d14-836f-d744209b4171-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.291274 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.291124    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.291274 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.291147    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crpkn\" (UniqueName: \"kubernetes.io/projected/6ddb5009-d762-4d14-836f-d744209b4171-kube-api-access-crpkn\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.291274 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.291217    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.291838 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.291809    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddb5009-d762-4d14-836f-d744209b4171-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.293632 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.293613    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.293710 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.293652    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.299103 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.299082    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crpkn\" (UniqueName: \"kubernetes.io/projected/6ddb5009-d762-4d14-836f-d744209b4171-kube-api-access-crpkn\") pod \"maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.429967 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.429900    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:36.710121 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.710095    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"]
Sep 08 16:43:36.712202 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:43:36.712178    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ddb5009_d762_4d14_836f_d744209b4171.slice/crio-7a0f0a8b6a96cf9f84f174fee75fad2b2d9d16002a828f77793e185542627d6c WatchSource:0}: Error finding container 7a0f0a8b6a96cf9f84f174fee75fad2b2d9d16002a828f77793e185542627d6c: Status 404 returned error can't find the container with id 7a0f0a8b6a96cf9f84f174fee75fad2b2d9d16002a828f77793e185542627d6c
Sep 08 16:43:36.785505 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.785473    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"]
Sep 08 16:43:36.790335 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.790315    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:36.793149 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.793129    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-8dca27\""
Sep 08 16:43:36.793248 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.793201    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-8dca27-dockercfg-t44dd\""
Sep 08 16:43:36.803825 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.803804    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"]
Sep 08 16:43:36.897161 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.897131    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4\" (UID: \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:36.897292 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.897178    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f4ms\" (UniqueName: \"kubernetes.io/projected/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-kube-api-access-5f4ms\") pod \"payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4\" (UID: \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:36.998347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.998274    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f4ms\" (UniqueName: \"kubernetes.io/projected/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-kube-api-access-5f4ms\") pod \"payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4\" (UID: \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:36.998468 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.998368    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4\" (UID: \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:36.999007 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:36.998988    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4\" (UID: \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:37.006977 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.006959    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f4ms\" (UniqueName: \"kubernetes.io/projected/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-kube-api-access-5f4ms\") pod \"payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4\" (UID: \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:37.100796 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.100768    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:37.372269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.372247    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"]
Sep 08 16:43:37.374182 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:43:37.374155    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ad94bb_67ab_43e8_b57c_5ab35c74f590.slice/crio-1ef250d6dbdadbb70f1ac1c54f36e7a38a8789ce850162cfa62d51082eb936e0 WatchSource:0}: Error finding container 1ef250d6dbdadbb70f1ac1c54f36e7a38a8789ce850162cfa62d51082eb936e0: Status 404 returned error can't find the container with id 1ef250d6dbdadbb70f1ac1c54f36e7a38a8789ce850162cfa62d51082eb936e0
Sep 08 16:43:37.615817 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.615781    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75" event={"ID":"6ddb5009-d762-4d14-836f-d744209b4171","Type":"ContainerStarted","Data":"cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a"}
Sep 08 16:43:37.615987 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.615844    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75" event={"ID":"6ddb5009-d762-4d14-836f-d744209b4171","Type":"ContainerStarted","Data":"7a0f0a8b6a96cf9f84f174fee75fad2b2d9d16002a828f77793e185542627d6c"}
Sep 08 16:43:37.616052 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.616007    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:37.618219 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.618189    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4" event={"ID":"f9ad94bb-67ab-43e8-b57c-5ab35c74f590","Type":"ContainerStarted","Data":"a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649"}
Sep 08 16:43:37.618351 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.618238    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4" event={"ID":"f9ad94bb-67ab-43e8-b57c-5ab35c74f590","Type":"ContainerStarted","Data":"1ef250d6dbdadbb70f1ac1c54f36e7a38a8789ce850162cfa62d51082eb936e0"}
Sep 08 16:43:37.618351 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.618340    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:43:37.638994 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.638933    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75" podStartSLOduration=1.083218105 podStartE2EDuration="1.63891448s" totalImagesPullingTime="555.696375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:37.636602874 +0000 UTC m=+1851.723266622" watchObservedRunningTime="2026-09-08 16:43:37.63891448 +0000 UTC m=+1851.725578232"
Sep 08 16:43:37.656246 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:37.656204    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4" podStartSLOduration=1.656191513 podStartE2EDuration="1.656191513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:43:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:43:37.654248981 +0000 UTC m=+1851.740912729" watchObservedRunningTime="2026-09-08 16:43:37.656191513 +0000 UTC m=+1851.742855261"
Sep 08 16:43:43.629340 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:43.629305    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:43:47.442898 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:47.442865    2575 scope.go:117] "RemoveContainer" containerID="3a188e68910bb16dc53b42d3f232112382c66060f53e40a8f999131bf4ff367d"
Sep 08 16:43:48.626641 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:43:48.626612    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:44:16.877524 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:16.877492    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"]
Sep 08 16:44:16.878131 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:16.877811    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75" podUID="6ddb5009-d762-4d14-836f-d744209b4171" containerName="maas-api" containerID="cri-o://cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a" gracePeriod=30
Sep 08 16:44:16.947306 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:16.947272    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"]
Sep 08 16:44:16.947549 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:16.947505    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4" podUID="f9ad94bb-67ab-43e8-b57c-5ab35c74f590" containerName="payload-pre-processing" containerID="cri-o://a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649" gracePeriod=30
Sep 08 16:44:17.221734 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.221704    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:44:17.248192 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.248163    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:44:17.259335 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.259313    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-metrics-certs\") pod \"6ddb5009-d762-4d14-836f-d744209b4171\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") "
Sep 08 16:44:17.259458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.259409    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddb5009-d762-4d14-836f-d744209b4171-openshift-service-ca\") pod \"6ddb5009-d762-4d14-836f-d744209b4171\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") "
Sep 08 16:44:17.259458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.259439    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crpkn\" (UniqueName: \"kubernetes.io/projected/6ddb5009-d762-4d14-836f-d744209b4171-kube-api-access-crpkn\") pod \"6ddb5009-d762-4d14-836f-d744209b4171\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") "
Sep 08 16:44:17.259573 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.259458    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-maas-api-tls\") pod \"6ddb5009-d762-4d14-836f-d744209b4171\" (UID: \"6ddb5009-d762-4d14-836f-d744209b4171\") "
Sep 08 16:44:17.259813 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.259785    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ddb5009-d762-4d14-836f-d744209b4171-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6ddb5009-d762-4d14-836f-d744209b4171" (UID: "6ddb5009-d762-4d14-836f-d744209b4171"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:17.261738 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.261716    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ddb5009-d762-4d14-836f-d744209b4171-kube-api-access-crpkn" (OuterVolumeSpecName: "kube-api-access-crpkn") pod "6ddb5009-d762-4d14-836f-d744209b4171" (UID: "6ddb5009-d762-4d14-836f-d744209b4171"). InnerVolumeSpecName "kube-api-access-crpkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:17.261823 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.261772    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6ddb5009-d762-4d14-836f-d744209b4171" (UID: "6ddb5009-d762-4d14-836f-d744209b4171"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:44:17.261823 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.261782    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6ddb5009-d762-4d14-836f-d744209b4171" (UID: "6ddb5009-d762-4d14-836f-d744209b4171"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:44:17.360671 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.360641    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-plugins-config-volume\") pod \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\" (UID: \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\") "
Sep 08 16:44:17.360840 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.360688    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f4ms\" (UniqueName: \"kubernetes.io/projected/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-kube-api-access-5f4ms\") pod \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\" (UID: \"f9ad94bb-67ab-43e8-b57c-5ab35c74f590\") "
Sep 08 16:44:17.360945 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.360930    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ddb5009-d762-4d14-836f-d744209b4171-openshift-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:17.360945 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.360944    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crpkn\" (UniqueName: \"kubernetes.io/projected/6ddb5009-d762-4d14-836f-d744209b4171-kube-api-access-crpkn\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:17.361071 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.360953    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-maas-api-tls\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:17.361071 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.360962    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ddb5009-d762-4d14-836f-d744209b4171-metrics-certs\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:17.361149 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.361064    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f9ad94bb-67ab-43e8-b57c-5ab35c74f590" (UID: "f9ad94bb-67ab-43e8-b57c-5ab35c74f590"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:17.362756 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.362736    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-kube-api-access-5f4ms" (OuterVolumeSpecName: "kube-api-access-5f4ms") pod "f9ad94bb-67ab-43e8-b57c-5ab35c74f590" (UID: "f9ad94bb-67ab-43e8-b57c-5ab35c74f590"). InnerVolumeSpecName "kube-api-access-5f4ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:17.461882 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.461815    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f4ms\" (UniqueName: \"kubernetes.io/projected/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-kube-api-access-5f4ms\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:17.461882 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.461840    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9ad94bb-67ab-43e8-b57c-5ab35c74f590-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:17.836240 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.836206    2575 generic.go:356] "Generic (PLEG): container finished" podID="f9ad94bb-67ab-43e8-b57c-5ab35c74f590" containerID="a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649" exitCode=0
Sep 08 16:44:17.836412 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.836253    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"
Sep 08 16:44:17.836412 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.836289    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4" event={"ID":"f9ad94bb-67ab-43e8-b57c-5ab35c74f590","Type":"ContainerDied","Data":"a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649"}
Sep 08 16:44:17.836412 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.836341    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4" event={"ID":"f9ad94bb-67ab-43e8-b57c-5ab35c74f590","Type":"ContainerDied","Data":"1ef250d6dbdadbb70f1ac1c54f36e7a38a8789ce850162cfa62d51082eb936e0"}
Sep 08 16:44:17.836412 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.836372    2575 scope.go:117] "RemoveContainer" containerID="a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649"
Sep 08 16:44:17.837724 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.837703    2575 generic.go:356] "Generic (PLEG): container finished" podID="6ddb5009-d762-4d14-836f-d744209b4171" containerID="cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a" exitCode=0
Sep 08 16:44:17.837804 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.837739    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"
Sep 08 16:44:17.837804 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.837775    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75" event={"ID":"6ddb5009-d762-4d14-836f-d744209b4171","Type":"ContainerDied","Data":"cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a"}
Sep 08 16:44:17.837874 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.837819    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75" event={"ID":"6ddb5009-d762-4d14-836f-d744209b4171","Type":"ContainerDied","Data":"7a0f0a8b6a96cf9f84f174fee75fad2b2d9d16002a828f77793e185542627d6c"}
Sep 08 16:44:17.846333 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.846312    2575 scope.go:117] "RemoveContainer" containerID="a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649"
Sep 08 16:44:17.846665 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:44:17.846642    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649\": container with ID starting with a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649 not found: ID does not exist" containerID="a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649"
Sep 08 16:44:17.846757 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.846670    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649"} err="failed to get container status \"a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649\": rpc error: code = NotFound desc = could not find container \"a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649\": container with ID starting with a2eb7dd5a932cf275f8d64ed2d256dca6b249a499dd708b2c09a3cdea973e649 not found: ID does not exist"
Sep 08 16:44:17.846757 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.846688    2575 scope.go:117] "RemoveContainer" containerID="cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a"
Sep 08 16:44:17.855253 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.855238    2575 scope.go:117] "RemoveContainer" containerID="cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a"
Sep 08 16:44:17.855484 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:44:17.855465    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a\": container with ID starting with cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a not found: ID does not exist" containerID="cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a"
Sep 08 16:44:17.855528 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.855490    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a"} err="failed to get container status \"cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a\": rpc error: code = NotFound desc = could not find container \"cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a\": container with ID starting with cc2c391773a556f466fe067c2d4509ed1fabae6f5ff69a19388a1e2c6257325a not found: ID does not exist"
Sep 08 16:44:17.862638 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.862616    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"]
Sep 08 16:44:17.867028 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.867008    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-8dca27-7dfcd75998-nmnc4"]
Sep 08 16:44:17.880824 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.880802    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"]
Sep 08 16:44:17.883902 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:17.883878    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-8dca27-84594cc878-6mb75"]
Sep 08 16:44:18.507125 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:18.507095    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ddb5009-d762-4d14-836f-d744209b4171" path="/var/lib/kubelet/pods/6ddb5009-d762-4d14-836f-d744209b4171/volumes"
Sep 08 16:44:18.507458 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:18.507445    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ad94bb-67ab-43e8-b57c-5ab35c74f590" path="/var/lib/kubelet/pods/f9ad94bb-67ab-43e8-b57c-5ab35c74f590/volumes"
Sep 08 16:44:27.341650 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.341615    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"]
Sep 08 16:44:27.342279 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.342261    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ddb5009-d762-4d14-836f-d744209b4171" containerName="maas-api"
Sep 08 16:44:27.342324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.342280    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ddb5009-d762-4d14-836f-d744209b4171" containerName="maas-api"
Sep 08 16:44:27.342324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.342298    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ad94bb-67ab-43e8-b57c-5ab35c74f590" containerName="payload-pre-processing"
Sep 08 16:44:27.342324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.342304    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ad94bb-67ab-43e8-b57c-5ab35c74f590" containerName="payload-pre-processing"
Sep 08 16:44:27.342416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.342375    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ad94bb-67ab-43e8-b57c-5ab35c74f590" containerName="payload-pre-processing"
Sep 08 16:44:27.342416 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.342384    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ddb5009-d762-4d14-836f-d744209b4171" containerName="maas-api"
Sep 08 16:44:27.345498 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.345483    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"
Sep 08 16:44:27.348291 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.348274    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:44:27.353312 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.353281    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"]
Sep 08 16:44:27.448687 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.448648    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec9434d-2d8b-4282-a958-2ae57bd6ae67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf\" (UID: \"6ec9434d-2d8b-4282-a958-2ae57bd6ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"
Sep 08 16:44:27.550085 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.550055    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec9434d-2d8b-4282-a958-2ae57bd6ae67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf\" (UID: \"6ec9434d-2d8b-4282-a958-2ae57bd6ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"
Sep 08 16:44:27.550847 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.550823    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec9434d-2d8b-4282-a958-2ae57bd6ae67-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf\" (UID: \"6ec9434d-2d8b-4282-a958-2ae57bd6ae67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"
Sep 08 16:44:27.657007 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.656943    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"
Sep 08 16:44:27.931144 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:27.931117    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"]
Sep 08 16:44:27.932988 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:44:27.932960    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ec9434d_2d8b_4282_a958_2ae57bd6ae67.slice/crio-695d2904ee74959dcf9391eab39ab00cafcb99a7d0aa6903681a0603a7f26b3e WatchSource:0}: Error finding container 695d2904ee74959dcf9391eab39ab00cafcb99a7d0aa6903681a0603a7f26b3e: Status 404 returned error can't find the container with id 695d2904ee74959dcf9391eab39ab00cafcb99a7d0aa6903681a0603a7f26b3e
Sep 08 16:44:28.905555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:28.905482    2575 generic.go:356] "Generic (PLEG): container finished" podID="6ec9434d-2d8b-4282-a958-2ae57bd6ae67" containerID="8a1c470d6a3860c64b0a6f19d8c833346aae7f223baf19d67e665a47cb3a2f65" exitCode=0
Sep 08 16:44:28.905555 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:28.905536    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf" event={"ID":"6ec9434d-2d8b-4282-a958-2ae57bd6ae67","Type":"ContainerDied","Data":"8a1c470d6a3860c64b0a6f19d8c833346aae7f223baf19d67e665a47cb3a2f65"}
Sep 08 16:44:28.905916 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:28.905573    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf" event={"ID":"6ec9434d-2d8b-4282-a958-2ae57bd6ae67","Type":"ContainerStarted","Data":"695d2904ee74959dcf9391eab39ab00cafcb99a7d0aa6903681a0603a7f26b3e"}
Sep 08 16:44:30.057067 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:30.057046    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"
Sep 08 16:44:30.172341 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:30.172266    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec9434d-2d8b-4282-a958-2ae57bd6ae67-maas-api-service-ca\") pod \"6ec9434d-2d8b-4282-a958-2ae57bd6ae67\" (UID: \"6ec9434d-2d8b-4282-a958-2ae57bd6ae67\") "
Sep 08 16:44:30.172618 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:30.172571    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec9434d-2d8b-4282-a958-2ae57bd6ae67-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ec9434d-2d8b-4282-a958-2ae57bd6ae67" (UID: "6ec9434d-2d8b-4282-a958-2ae57bd6ae67"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:30.273669 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:30.273645    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ec9434d-2d8b-4282-a958-2ae57bd6ae67-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:30.917466 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:30.917432    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf" event={"ID":"6ec9434d-2d8b-4282-a958-2ae57bd6ae67","Type":"ContainerDied","Data":"695d2904ee74959dcf9391eab39ab00cafcb99a7d0aa6903681a0603a7f26b3e"}
Sep 08 16:44:30.917647 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:30.917480    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="695d2904ee74959dcf9391eab39ab00cafcb99a7d0aa6903681a0603a7f26b3e"
Sep 08 16:44:30.917647 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:30.917439    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"
Sep 08 16:44:42.973127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:42.973093    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"]
Sep 08 16:44:42.973613 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:42.973310    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv" podUID="27cb0dc4-2d67-4cef-bbcc-8105480ad028" containerName="payload-pre-processing" containerID="cri-o://f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641" gracePeriod=30
Sep 08 16:44:43.188486 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.188456    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"]
Sep 08 16:44:43.192737 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.192713    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-352142-f5kcf"]
Sep 08 16:44:43.228965 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.228917    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:44:43.289862 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.289830    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cb0dc4-2d67-4cef-bbcc-8105480ad028-plugins-config-volume\") pod \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\" (UID: \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\") "
Sep 08 16:44:43.290013 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.289939    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q27x\" (UniqueName: \"kubernetes.io/projected/27cb0dc4-2d67-4cef-bbcc-8105480ad028-kube-api-access-8q27x\") pod \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\" (UID: \"27cb0dc4-2d67-4cef-bbcc-8105480ad028\") "
Sep 08 16:44:43.290250 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.290220    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27cb0dc4-2d67-4cef-bbcc-8105480ad028-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "27cb0dc4-2d67-4cef-bbcc-8105480ad028" (UID: "27cb0dc4-2d67-4cef-bbcc-8105480ad028"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:44:43.292073 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.292055    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27cb0dc4-2d67-4cef-bbcc-8105480ad028-kube-api-access-8q27x" (OuterVolumeSpecName: "kube-api-access-8q27x") pod "27cb0dc4-2d67-4cef-bbcc-8105480ad028" (UID: "27cb0dc4-2d67-4cef-bbcc-8105480ad028"). InnerVolumeSpecName "kube-api-access-8q27x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:44:43.390970 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.390941    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q27x\" (UniqueName: \"kubernetes.io/projected/27cb0dc4-2d67-4cef-bbcc-8105480ad028-kube-api-access-8q27x\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:43.390970 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.390965    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27cb0dc4-2d67-4cef-bbcc-8105480ad028-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:44:43.989154 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.989118    2575 generic.go:356] "Generic (PLEG): container finished" podID="27cb0dc4-2d67-4cef-bbcc-8105480ad028" containerID="f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641" exitCode=0
Sep 08 16:44:43.989625 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.989166    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"
Sep 08 16:44:43.989625 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.989213    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv" event={"ID":"27cb0dc4-2d67-4cef-bbcc-8105480ad028","Type":"ContainerDied","Data":"f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641"}
Sep 08 16:44:43.989625 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.989277    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv" event={"ID":"27cb0dc4-2d67-4cef-bbcc-8105480ad028","Type":"ContainerDied","Data":"85f5266430e97b57f317ac715c1796cd5ef3506d728f9a4349e1f92c77220c0a"}
Sep 08 16:44:43.989625 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.989310    2575 scope.go:117] "RemoveContainer" containerID="f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641"
Sep 08 16:44:43.998111 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.998091    2575 scope.go:117] "RemoveContainer" containerID="f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641"
Sep 08 16:44:43.998375 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:44:43.998354    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641\": container with ID starting with f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641 not found: ID does not exist" containerID="f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641"
Sep 08 16:44:43.998422 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:43.998384    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641"} err="failed to get container status \"f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641\": rpc error: code = NotFound desc = could not find container \"f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641\": container with ID starting with f447b1ee5e9b502893ea9ad1ef641e9c2cb0210e60a0523869f27fa89d10a641 not found: ID does not exist"
Sep 08 16:44:44.021681 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:44.021659    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"]
Sep 08 16:44:44.026003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:44.025983    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-352142-586c95d466-955mv"]
Sep 08 16:44:44.506958 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:44.506925    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27cb0dc4-2d67-4cef-bbcc-8105480ad028" path="/var/lib/kubelet/pods/27cb0dc4-2d67-4cef-bbcc-8105480ad028/volumes"
Sep 08 16:44:44.507244 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:44:44.507231    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ec9434d-2d8b-4282-a958-2ae57bd6ae67" path="/var/lib/kubelet/pods/6ec9434d-2d8b-4282-a958-2ae57bd6ae67/volumes"
Sep 08 16:45:00.179163 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.179127    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"]
Sep 08 16:45:00.179627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.179539    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27cb0dc4-2d67-4cef-bbcc-8105480ad028" containerName="payload-pre-processing"
Sep 08 16:45:00.179627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.179553    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="27cb0dc4-2d67-4cef-bbcc-8105480ad028" containerName="payload-pre-processing"
Sep 08 16:45:00.179627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.179561    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ec9434d-2d8b-4282-a958-2ae57bd6ae67" containerName="revoke-keys"
Sep 08 16:45:00.179627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.179566    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ec9434d-2d8b-4282-a958-2ae57bd6ae67" containerName="revoke-keys"
Sep 08 16:45:00.179766 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.179653    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ec9434d-2d8b-4282-a958-2ae57bd6ae67" containerName="revoke-keys"
Sep 08 16:45:00.179766 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.179663    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="27cb0dc4-2d67-4cef-bbcc-8105480ad028" containerName="payload-pre-processing"
Sep 08 16:45:00.182934 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.182913    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"]
Sep 08 16:45:00.183068 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.183049    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"
Sep 08 16:45:00.186550 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.186529    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"
Sep 08 16:45:00.192123 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.192101    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:45:00.195240 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.195224    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"
Sep 08 16:45:00.202391 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.202369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"
Sep 08 16:45:00.204766 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.204682    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"]
Sep 08 16:45:00.210896 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.208046    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"]
Sep 08 16:45:00.492492 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.492448    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"]
Sep 08 16:45:00.493931 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:45:00.493900    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd6c83c_c3d3_46d2_acd0_cc3478927b6b.slice/crio-5b25207662013cf724f58a20d77f0bbdf9e3fa07d01fbc59c0f0b674a704861c WatchSource:0}: Error finding container 5b25207662013cf724f58a20d77f0bbdf9e3fa07d01fbc59c0f0b674a704861c: Status 404 returned error can't find the container with id 5b25207662013cf724f58a20d77f0bbdf9e3fa07d01fbc59c0f0b674a704861c
Sep 08 16:45:00.494747 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:00.494721    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"]
Sep 08 16:45:00.497424 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:45:00.497402    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fcbe89f_ccc1_4751_a358_1a44be3db35a.slice/crio-3248e5ff6b163c5f128e90aad1249ba7985a6cdb3b17808f70cd10fd7d225782 WatchSource:0}: Error finding container 3248e5ff6b163c5f128e90aad1249ba7985a6cdb3b17808f70cd10fd7d225782: Status 404 returned error can't find the container with id 3248e5ff6b163c5f128e90aad1249ba7985a6cdb3b17808f70cd10fd7d225782
Sep 08 16:45:01.085702 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:01.085605    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4" event={"ID":"5bd6c83c-c3d3-46d2-acd0-cc3478927b6b","Type":"ContainerStarted","Data":"06dad105d213fb99fa7ab9ac6434a3ceb82792eb9ad10ab56b25f0826f776759"}
Sep 08 16:45:01.085702 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:01.085673    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4" event={"ID":"5bd6c83c-c3d3-46d2-acd0-cc3478927b6b","Type":"ContainerStarted","Data":"5b25207662013cf724f58a20d77f0bbdf9e3fa07d01fbc59c0f0b674a704861c"}
Sep 08 16:45:01.087274 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:01.087249    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2" event={"ID":"7fcbe89f-ccc1-4751-a358-1a44be3db35a","Type":"ContainerStarted","Data":"ea5dbdffc5740b8bc3ef471c5a5fdc6dbb87a203d17630f694610ee514b29804"}
Sep 08 16:45:01.087377 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:01.087289    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2" event={"ID":"7fcbe89f-ccc1-4751-a358-1a44be3db35a","Type":"ContainerStarted","Data":"3248e5ff6b163c5f128e90aad1249ba7985a6cdb3b17808f70cd10fd7d225782"}
Sep 08 16:45:01.104958 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:01.104906    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4" podStartSLOduration=1.104887827 podStartE2EDuration="1.104887827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:01.104341739 +0000 UTC m=+1935.191005489" watchObservedRunningTime="2026-09-08 16:45:01.104887827 +0000 UTC m=+1935.191551575"
Sep 08 16:45:01.123787 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:01.123740    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2" podStartSLOduration=1.123723531 podStartE2EDuration="1.123723531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:45:01.121257168 +0000 UTC m=+1935.207920944" watchObservedRunningTime="2026-09-08 16:45:01.123723531 +0000 UTC m=+1935.210387279"
Sep 08 16:45:02.094205 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:02.094171    2575 generic.go:356] "Generic (PLEG): container finished" podID="7fcbe89f-ccc1-4751-a358-1a44be3db35a" containerID="ea5dbdffc5740b8bc3ef471c5a5fdc6dbb87a203d17630f694610ee514b29804" exitCode=0
Sep 08 16:45:02.094631 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:02.094256    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2" event={"ID":"7fcbe89f-ccc1-4751-a358-1a44be3db35a","Type":"ContainerDied","Data":"ea5dbdffc5740b8bc3ef471c5a5fdc6dbb87a203d17630f694610ee514b29804"}
Sep 08 16:45:02.095791 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:02.095770    2575 generic.go:356] "Generic (PLEG): container finished" podID="5bd6c83c-c3d3-46d2-acd0-cc3478927b6b" containerID="06dad105d213fb99fa7ab9ac6434a3ceb82792eb9ad10ab56b25f0826f776759" exitCode=0
Sep 08 16:45:02.095886 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:02.095811    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4" event={"ID":"5bd6c83c-c3d3-46d2-acd0-cc3478927b6b","Type":"ContainerDied","Data":"06dad105d213fb99fa7ab9ac6434a3ceb82792eb9ad10ab56b25f0826f776759"}
Sep 08 16:45:03.235442 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:03.235410    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"
Sep 08 16:45:03.276233 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:03.276208    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"
Sep 08 16:45:04.108834 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:04.108803    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"
Sep 08 16:45:04.108834 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:04.108824    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4" event={"ID":"5bd6c83c-c3d3-46d2-acd0-cc3478927b6b","Type":"ContainerDied","Data":"5b25207662013cf724f58a20d77f0bbdf9e3fa07d01fbc59c0f0b674a704861c"}
Sep 08 16:45:04.109100 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:04.108873    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b25207662013cf724f58a20d77f0bbdf9e3fa07d01fbc59c0f0b674a704861c"
Sep 08 16:45:04.110117 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:04.110098    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"
Sep 08 16:45:04.110227 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:04.110126    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2" event={"ID":"7fcbe89f-ccc1-4751-a358-1a44be3db35a","Type":"ContainerDied","Data":"3248e5ff6b163c5f128e90aad1249ba7985a6cdb3b17808f70cd10fd7d225782"}
Sep 08 16:45:04.110227 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:04.110167    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3248e5ff6b163c5f128e90aad1249ba7985a6cdb3b17808f70cd10fd7d225782"
Sep 08 16:45:48.474519 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.474474    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"]
Sep 08 16:45:48.475035 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.474784    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s" podUID="784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" containerName="maas-api" containerID="cri-o://30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39" gracePeriod=30
Sep 08 16:45:48.507881 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.507855    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"]
Sep 08 16:45:48.507979 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.507889    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-995b22-29814765-nvbn4"]
Sep 08 16:45:48.531547 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.531515    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"]
Sep 08 16:45:48.531830 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.531775    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r" podUID="6cb96ee5-b26d-4983-82a6-9cd05241ac2d" containerName="payload-processing" containerID="cri-o://492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5" gracePeriod=30
Sep 08 16:45:48.544900 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.544869    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"]
Sep 08 16:45:48.545331 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.545279    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2" podUID="ed129475-b937-44a7-8476-0c4cdf51366d" containerName="payload-pre-processing" containerID="cri-o://eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4" gracePeriod=30
Sep 08 16:45:48.838058 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.838033    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:45:48.842351 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.842332    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:45:48.855828 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.855803    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:45:48.937901 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.937870    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls\") pod \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") "
Sep 08 16:45:48.938052 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.937959    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st4ms\" (UniqueName: \"kubernetes.io/projected/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-kube-api-access-st4ms\") pod \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\" (UID: \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\") "
Sep 08 16:45:48.938052 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938018    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-plugins-config-volume\") pod \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\" (UID: \"6cb96ee5-b26d-4983-82a6-9cd05241ac2d\") "
Sep 08 16:45:48.938137 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938054    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-openshift-service-ca\") pod \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") "
Sep 08 16:45:48.938137 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938085    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/ed129475-b937-44a7-8476-0c4cdf51366d-kube-api-access-s6nhr\") pod \"ed129475-b937-44a7-8476-0c4cdf51366d\" (UID: \"ed129475-b937-44a7-8476-0c4cdf51366d\") "
Sep 08 16:45:48.938246 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938214    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed129475-b937-44a7-8476-0c4cdf51366d-plugins-config-volume\") pod \"ed129475-b937-44a7-8476-0c4cdf51366d\" (UID: \"ed129475-b937-44a7-8476-0c4cdf51366d\") "
Sep 08 16:45:48.938300 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938283    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-metrics-certs\") pod \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") "
Sep 08 16:45:48.938353 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938320    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vvwq\" (UniqueName: \"kubernetes.io/projected/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-kube-api-access-9vvwq\") pod \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\" (UID: \"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3\") "
Sep 08 16:45:48.938606 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938485    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" (UID: "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:48.938606 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938558    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6cb96ee5-b26d-4983-82a6-9cd05241ac2d" (UID: "6cb96ee5-b26d-4983-82a6-9cd05241ac2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:48.938606 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938594    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed129475-b937-44a7-8476-0c4cdf51366d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed129475-b937-44a7-8476-0c4cdf51366d" (UID: "ed129475-b937-44a7-8476-0c4cdf51366d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:48.938835 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938690    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:48.938835 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938710    2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-openshift-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:48.938835 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.938723    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed129475-b937-44a7-8476-0c4cdf51366d-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:48.940231 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.940205    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-kube-api-access-st4ms" (OuterVolumeSpecName: "kube-api-access-st4ms") pod "6cb96ee5-b26d-4983-82a6-9cd05241ac2d" (UID: "6cb96ee5-b26d-4983-82a6-9cd05241ac2d"). InnerVolumeSpecName "kube-api-access-st4ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:48.940432 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.940412    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-kube-api-access-9vvwq" (OuterVolumeSpecName: "kube-api-access-9vvwq") pod "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" (UID: "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3"). InnerVolumeSpecName "kube-api-access-9vvwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:48.940538 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.940520    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" (UID: "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:45:48.940697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.940520    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed129475-b937-44a7-8476-0c4cdf51366d-kube-api-access-s6nhr" (OuterVolumeSpecName: "kube-api-access-s6nhr") pod "ed129475-b937-44a7-8476-0c4cdf51366d" (UID: "ed129475-b937-44a7-8476-0c4cdf51366d"). InnerVolumeSpecName "kube-api-access-s6nhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:45:48.940697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:48.940547    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" (UID: "784aac9b-2f12-4ce5-91c7-0fba9e40bcc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 08 16:45:49.039725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.039641    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6nhr\" (UniqueName: \"kubernetes.io/projected/ed129475-b937-44a7-8476-0c4cdf51366d-kube-api-access-s6nhr\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:49.039725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.039674    2575 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-metrics-certs\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:49.039725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.039686    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vvwq\" (UniqueName: \"kubernetes.io/projected/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-kube-api-access-9vvwq\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:49.039725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.039697    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3-maas-api-tls\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:49.039725 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.039711    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4ms\" (UniqueName: \"kubernetes.io/projected/6cb96ee5-b26d-4983-82a6-9cd05241ac2d-kube-api-access-st4ms\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:49.354034 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.353998    2575 generic.go:356] "Generic (PLEG): container finished" podID="6cb96ee5-b26d-4983-82a6-9cd05241ac2d" containerID="492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5" exitCode=0
Sep 08 16:45:49.354247 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.354049    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"
Sep 08 16:45:49.354247 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.354092    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r" event={"ID":"6cb96ee5-b26d-4983-82a6-9cd05241ac2d","Type":"ContainerDied","Data":"492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5"}
Sep 08 16:45:49.354247 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.354144    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r" event={"ID":"6cb96ee5-b26d-4983-82a6-9cd05241ac2d","Type":"ContainerDied","Data":"57275270f30d17e62dcf81b3639ebdef29c26a5d628ee28dd866e5f54a4ef44b"}
Sep 08 16:45:49.354247 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.354172    2575 scope.go:117] "RemoveContainer" containerID="492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5"
Sep 08 16:45:49.355592 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.355558    2575 generic.go:356] "Generic (PLEG): container finished" podID="ed129475-b937-44a7-8476-0c4cdf51366d" containerID="eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4" exitCode=0
Sep 08 16:45:49.355667 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.355618    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"
Sep 08 16:45:49.355742 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.355616    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2" event={"ID":"ed129475-b937-44a7-8476-0c4cdf51366d","Type":"ContainerDied","Data":"eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4"}
Sep 08 16:45:49.355805 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.355782    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2" event={"ID":"ed129475-b937-44a7-8476-0c4cdf51366d","Type":"ContainerDied","Data":"e0c16a11f84e79584367bcbe5450ea8292e6e1f8516aa7e50167bd89ec09b5d0"}
Sep 08 16:45:49.357155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.357133    2575 generic.go:356] "Generic (PLEG): container finished" podID="784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" containerID="30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39" exitCode=0
Sep 08 16:45:49.357269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.357181    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s" event={"ID":"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3","Type":"ContainerDied","Data":"30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39"}
Sep 08 16:45:49.357269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.357223    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s" event={"ID":"784aac9b-2f12-4ce5-91c7-0fba9e40bcc3","Type":"ContainerDied","Data":"4f3e41bbfd4b99693919998a55aca79161cb61d1967ef24b3942872e49a3c895"}
Sep 08 16:45:49.357269 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.357191    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"
Sep 08 16:45:49.364926 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.364907    2575 scope.go:117] "RemoveContainer" containerID="492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5"
Sep 08 16:45:49.365247 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:45:49.365223    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5\": container with ID starting with 492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5 not found: ID does not exist" containerID="492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5"
Sep 08 16:45:49.365349 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.365254    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5"} err="failed to get container status \"492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5\": rpc error: code = NotFound desc = could not find container \"492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5\": container with ID starting with 492301f1ea7117b90a4dcdbe49961c0bde3ffc40aca57be01ee1648085298fc5 not found: ID does not exist"
Sep 08 16:45:49.365349 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.365272    2575 scope.go:117] "RemoveContainer" containerID="eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4"
Sep 08 16:45:49.373779 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.373744    2575 scope.go:117] "RemoveContainer" containerID="eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4"
Sep 08 16:45:49.374082 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:45:49.374060    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4\": container with ID starting with eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4 not found: ID does not exist" containerID="eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4"
Sep 08 16:45:49.374127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.374091    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4"} err="failed to get container status \"eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4\": rpc error: code = NotFound desc = could not find container \"eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4\": container with ID starting with eed852ef18a13dde25617498e4f6cf039d8b1109a24afd81b92bf42e6fab0ba4 not found: ID does not exist"
Sep 08 16:45:49.374127 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.374110    2575 scope.go:117] "RemoveContainer" containerID="30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39"
Sep 08 16:45:49.382619 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.382567    2575 scope.go:117] "RemoveContainer" containerID="30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39"
Sep 08 16:45:49.382999 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:45:49.382976    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39\": container with ID starting with 30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39 not found: ID does not exist" containerID="30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39"
Sep 08 16:45:49.383090 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.383005    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39"} err="failed to get container status \"30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39\": rpc error: code = NotFound desc = could not find container \"30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39\": container with ID starting with 30d9d376d01b0ccf7e5456d397c0d4fb52621718d046f1a2a2c7e360116d0d39 not found: ID does not exist"
Sep 08 16:45:49.390723 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.390690    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"]
Sep 08 16:45:49.401074 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.401044    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-995b22-8568ccc7d5-dmv6r"]
Sep 08 16:45:49.419697 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.419659    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"]
Sep 08 16:45:49.424473 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.424442    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-995b22-586f9c6546-jxz9s"]
Sep 08 16:45:49.439281 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.439246    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"]
Sep 08 16:45:49.441763 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:49.441738    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-995b22-778bb956c8-hlkp2"]
Sep 08 16:45:50.506241 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:50.506208    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd6c83c-c3d3-46d2-acd0-cc3478927b6b" path="/var/lib/kubelet/pods/5bd6c83c-c3d3-46d2-acd0-cc3478927b6b/volumes"
Sep 08 16:45:50.506640 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:50.506536    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6cb96ee5-b26d-4983-82a6-9cd05241ac2d" path="/var/lib/kubelet/pods/6cb96ee5-b26d-4983-82a6-9cd05241ac2d/volumes"
Sep 08 16:45:50.506900 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:50.506886    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" path="/var/lib/kubelet/pods/784aac9b-2f12-4ce5-91c7-0fba9e40bcc3/volumes"
Sep 08 16:45:50.507195 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:50.507183    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed129475-b937-44a7-8476-0c4cdf51366d" path="/var/lib/kubelet/pods/ed129475-b937-44a7-8476-0c4cdf51366d/volumes"
Sep 08 16:45:55.600363 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600330    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"]
Sep 08 16:45:55.600745 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600722    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fcbe89f-ccc1-4751-a358-1a44be3db35a" containerName="cleanup"
Sep 08 16:45:55.600745 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600735    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fcbe89f-ccc1-4751-a358-1a44be3db35a" containerName="cleanup"
Sep 08 16:45:55.600745 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600745    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" containerName="maas-api"
Sep 08 16:45:55.600852 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600750    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" containerName="maas-api"
Sep 08 16:45:55.600852 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600765    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd6c83c-c3d3-46d2-acd0-cc3478927b6b" containerName="cleanup"
Sep 08 16:45:55.600852 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600771    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd6c83c-c3d3-46d2-acd0-cc3478927b6b" containerName="cleanup"
Sep 08 16:45:55.600852 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600778    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed129475-b937-44a7-8476-0c4cdf51366d" containerName="payload-pre-processing"
Sep 08 16:45:55.600852 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600783    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed129475-b937-44a7-8476-0c4cdf51366d" containerName="payload-pre-processing"
Sep 08 16:45:55.600852 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600792    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cb96ee5-b26d-4983-82a6-9cd05241ac2d" containerName="payload-processing"
Sep 08 16:45:55.600852 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600797    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cb96ee5-b26d-4983-82a6-9cd05241ac2d" containerName="payload-processing"
Sep 08 16:45:55.601056 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600859    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="784aac9b-2f12-4ce5-91c7-0fba9e40bcc3" containerName="maas-api"
Sep 08 16:45:55.601056 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600868    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cb96ee5-b26d-4983-82a6-9cd05241ac2d" containerName="payload-processing"
Sep 08 16:45:55.601056 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600875    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed129475-b937-44a7-8476-0c4cdf51366d" containerName="payload-pre-processing"
Sep 08 16:45:55.601056 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600881    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd6c83c-c3d3-46d2-acd0-cc3478927b6b" containerName="cleanup"
Sep 08 16:45:55.601056 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.600887    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fcbe89f-ccc1-4751-a358-1a44be3db35a" containerName="cleanup"
Sep 08 16:45:55.605641 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.605623    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"
Sep 08 16:45:55.607957 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.607935    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:45:55.611354 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.611332    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"]
Sep 08 16:45:55.697519 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.697494    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ee456c-6812-45a8-9133-200fee79492e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d\" (UID: \"46ee456c-6812-45a8-9133-200fee79492e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"
Sep 08 16:45:55.798312 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.798284    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ee456c-6812-45a8-9133-200fee79492e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d\" (UID: \"46ee456c-6812-45a8-9133-200fee79492e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"
Sep 08 16:45:55.798883 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.798866    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ee456c-6812-45a8-9133-200fee79492e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d\" (UID: \"46ee456c-6812-45a8-9133-200fee79492e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"
Sep 08 16:45:55.917336 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:55.917274    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"
Sep 08 16:45:56.190834 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:56.190808    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"]
Sep 08 16:45:56.193218 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:45:56.193189    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ee456c_6812_45a8_9133_200fee79492e.slice/crio-12a88929bc515537cb076ca14391348e2dea80c028908bb9c907f85450d3e50b WatchSource:0}: Error finding container 12a88929bc515537cb076ca14391348e2dea80c028908bb9c907f85450d3e50b: Status 404 returned error can't find the container with id 12a88929bc515537cb076ca14391348e2dea80c028908bb9c907f85450d3e50b
Sep 08 16:45:56.400942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:56.400906    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d" event={"ID":"46ee456c-6812-45a8-9133-200fee79492e","Type":"ContainerStarted","Data":"12a88929bc515537cb076ca14391348e2dea80c028908bb9c907f85450d3e50b"}
Sep 08 16:45:57.411186 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:57.411145    2575 generic.go:356] "Generic (PLEG): container finished" podID="46ee456c-6812-45a8-9133-200fee79492e" containerID="9657fde6293700319ba34549d5e8236869e3f690b552bd009a1686b7a8a92818" exitCode=0
Sep 08 16:45:57.411597 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:57.411234    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d" event={"ID":"46ee456c-6812-45a8-9133-200fee79492e","Type":"ContainerDied","Data":"9657fde6293700319ba34549d5e8236869e3f690b552bd009a1686b7a8a92818"}
Sep 08 16:45:58.563592 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:58.563565    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"
Sep 08 16:45:58.725120 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:58.725053    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ee456c-6812-45a8-9133-200fee79492e-maas-api-service-ca\") pod \"46ee456c-6812-45a8-9133-200fee79492e\" (UID: \"46ee456c-6812-45a8-9133-200fee79492e\") "
Sep 08 16:45:58.725406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:58.725386    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ee456c-6812-45a8-9133-200fee79492e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46ee456c-6812-45a8-9133-200fee79492e" (UID: "46ee456c-6812-45a8-9133-200fee79492e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:45:58.825927 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:58.825894    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46ee456c-6812-45a8-9133-200fee79492e-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:45:59.423197 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:59.423167    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"
Sep 08 16:45:59.423395 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:59.423170    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d" event={"ID":"46ee456c-6812-45a8-9133-200fee79492e","Type":"ContainerDied","Data":"12a88929bc515537cb076ca14391348e2dea80c028908bb9c907f85450d3e50b"}
Sep 08 16:45:59.423395 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:45:59.423291    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12a88929bc515537cb076ca14391348e2dea80c028908bb9c907f85450d3e50b"
Sep 08 16:46:06.256347 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.256310    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"]
Sep 08 16:46:06.256775 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.256512    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5" podUID="1c720582-2ec1-453e-9ffc-fdb120bd111e" containerName="payload-pre-processing" containerID="cri-o://732593dd20b76a93f422ace4c258010db45eedb53bca5d9c42c1f129b098e758" gracePeriod=30
Sep 08 16:46:06.420406 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.420340    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"]
Sep 08 16:46:06.423618 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.423573    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-bce20b-xh88d"]
Sep 08 16:46:06.462976 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.462941    2575 generic.go:356] "Generic (PLEG): container finished" podID="1c720582-2ec1-453e-9ffc-fdb120bd111e" containerID="732593dd20b76a93f422ace4c258010db45eedb53bca5d9c42c1f129b098e758" exitCode=0
Sep 08 16:46:06.463108 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.463009    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5" event={"ID":"1c720582-2ec1-453e-9ffc-fdb120bd111e","Type":"ContainerDied","Data":"732593dd20b76a93f422ace4c258010db45eedb53bca5d9c42c1f129b098e758"}
Sep 08 16:46:06.508267 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.508242    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46ee456c-6812-45a8-9133-200fee79492e" path="/var/lib/kubelet/pods/46ee456c-6812-45a8-9133-200fee79492e/volumes"
Sep 08 16:46:06.524987 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.524965    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:46:06.589129 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.589092    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c720582-2ec1-453e-9ffc-fdb120bd111e-plugins-config-volume\") pod \"1c720582-2ec1-453e-9ffc-fdb120bd111e\" (UID: \"1c720582-2ec1-453e-9ffc-fdb120bd111e\") "
Sep 08 16:46:06.589282 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.589155    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkqtm\" (UniqueName: \"kubernetes.io/projected/1c720582-2ec1-453e-9ffc-fdb120bd111e-kube-api-access-pkqtm\") pod \"1c720582-2ec1-453e-9ffc-fdb120bd111e\" (UID: \"1c720582-2ec1-453e-9ffc-fdb120bd111e\") "
Sep 08 16:46:06.589435 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.589410    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c720582-2ec1-453e-9ffc-fdb120bd111e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c720582-2ec1-453e-9ffc-fdb120bd111e" (UID: "1c720582-2ec1-453e-9ffc-fdb120bd111e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:06.591068 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.591043    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c720582-2ec1-453e-9ffc-fdb120bd111e-kube-api-access-pkqtm" (OuterVolumeSpecName: "kube-api-access-pkqtm") pod "1c720582-2ec1-453e-9ffc-fdb120bd111e" (UID: "1c720582-2ec1-453e-9ffc-fdb120bd111e"). InnerVolumeSpecName "kube-api-access-pkqtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:06.690230 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.690202    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c720582-2ec1-453e-9ffc-fdb120bd111e-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:46:06.690230 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:06.690226    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqtm\" (UniqueName: \"kubernetes.io/projected/1c720582-2ec1-453e-9ffc-fdb120bd111e-kube-api-access-pkqtm\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:46:07.469996 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:07.469962    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"
Sep 08 16:46:07.469996 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:07.469980    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5" event={"ID":"1c720582-2ec1-453e-9ffc-fdb120bd111e","Type":"ContainerDied","Data":"8b4facf49c1c527a10f453afa5ad0b7489722b985987eeba0a9e7dcd3ae86f1e"}
Sep 08 16:46:07.470508 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:07.470048    2575 scope.go:117] "RemoveContainer" containerID="732593dd20b76a93f422ace4c258010db45eedb53bca5d9c42c1f129b098e758"
Sep 08 16:46:07.511184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:07.511158    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"]
Sep 08 16:46:07.523605 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:07.522076    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-bce20b-f98cddf98-9p2k5"]
Sep 08 16:46:08.506773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:08.506738    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c720582-2ec1-453e-9ffc-fdb120bd111e" path="/var/lib/kubelet/pods/1c720582-2ec1-453e-9ffc-fdb120bd111e/volumes"
Sep 08 16:46:13.538044 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.538010    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"]
Sep 08 16:46:13.538415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.538384    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46ee456c-6812-45a8-9133-200fee79492e" containerName="revoke-keys"
Sep 08 16:46:13.538415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.538395    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46ee456c-6812-45a8-9133-200fee79492e" containerName="revoke-keys"
Sep 08 16:46:13.538415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.538405    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c720582-2ec1-453e-9ffc-fdb120bd111e" containerName="payload-pre-processing"
Sep 08 16:46:13.538415 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.538410    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c720582-2ec1-453e-9ffc-fdb120bd111e" containerName="payload-pre-processing"
Sep 08 16:46:13.538549 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.538476    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c720582-2ec1-453e-9ffc-fdb120bd111e" containerName="payload-pre-processing"
Sep 08 16:46:13.538549 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.538488    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46ee456c-6812-45a8-9133-200fee79492e" containerName="revoke-keys"
Sep 08 16:46:13.542784 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.542768    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"
Sep 08 16:46:13.546320 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.546298    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2qr4j\""
Sep 08 16:46:13.557912 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.557888    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"]
Sep 08 16:46:13.652904 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.652874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612c3829-11ce-4f72-8174-94eff2ec84c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658\" (UID: \"612c3829-11ce-4f72-8174-94eff2ec84c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"
Sep 08 16:46:13.754145 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.754109    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612c3829-11ce-4f72-8174-94eff2ec84c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658\" (UID: \"612c3829-11ce-4f72-8174-94eff2ec84c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"
Sep 08 16:46:13.754739 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.754721    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612c3829-11ce-4f72-8174-94eff2ec84c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658\" (UID: \"612c3829-11ce-4f72-8174-94eff2ec84c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"
Sep 08 16:46:13.852917 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:13.852889    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"
Sep 08 16:46:14.122644 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:14.122617    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"]
Sep 08 16:46:14.124739 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:46:14.124714    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612c3829_11ce_4f72_8174_94eff2ec84c9.slice/crio-5599e07e93bd0bd21a73574c52f271b8165b4822669f8ef25ea196f05151c565 WatchSource:0}: Error finding container 5599e07e93bd0bd21a73574c52f271b8165b4822669f8ef25ea196f05151c565: Status 404 returned error can't find the container with id 5599e07e93bd0bd21a73574c52f271b8165b4822669f8ef25ea196f05151c565
Sep 08 16:46:14.510085 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:14.510056    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658" event={"ID":"612c3829-11ce-4f72-8174-94eff2ec84c9","Type":"ContainerStarted","Data":"a0c39d001a1deb5fdc19c27d6f62834f14cca511ab70f1cc50760a49bac54c83"}
Sep 08 16:46:14.510191 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:14.510108    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658" event={"ID":"612c3829-11ce-4f72-8174-94eff2ec84c9","Type":"ContainerStarted","Data":"5599e07e93bd0bd21a73574c52f271b8165b4822669f8ef25ea196f05151c565"}
Sep 08 16:46:14.525146 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:14.525100    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658" podStartSLOduration=1.5250837480000001 podStartE2EDuration="1.525083748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:46:14.524213835 +0000 UTC m=+2008.610877586" watchObservedRunningTime="2026-09-08 16:46:14.525083748 +0000 UTC m=+2008.611747498"
Sep 08 16:46:15.516661 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:15.516624    2575 generic.go:356] "Generic (PLEG): container finished" podID="612c3829-11ce-4f72-8174-94eff2ec84c9" containerID="a0c39d001a1deb5fdc19c27d6f62834f14cca511ab70f1cc50760a49bac54c83" exitCode=0
Sep 08 16:46:15.517081 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:15.516688    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658" event={"ID":"612c3829-11ce-4f72-8174-94eff2ec84c9","Type":"ContainerDied","Data":"a0c39d001a1deb5fdc19c27d6f62834f14cca511ab70f1cc50760a49bac54c83"}
Sep 08 16:46:16.653479 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:16.653453    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"
Sep 08 16:46:16.679890 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:16.679865    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612c3829-11ce-4f72-8174-94eff2ec84c9-maas-api-service-ca\") pod \"612c3829-11ce-4f72-8174-94eff2ec84c9\" (UID: \"612c3829-11ce-4f72-8174-94eff2ec84c9\") "
Sep 08 16:46:16.680244 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:16.680218    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612c3829-11ce-4f72-8174-94eff2ec84c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "612c3829-11ce-4f72-8174-94eff2ec84c9" (UID: "612c3829-11ce-4f72-8174-94eff2ec84c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:16.781369 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:16.781310    2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612c3829-11ce-4f72-8174-94eff2ec84c9-maas-api-service-ca\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:46:17.527261 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:17.527227    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"
Sep 08 16:46:17.527426 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:17.527263    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658" event={"ID":"612c3829-11ce-4f72-8174-94eff2ec84c9","Type":"ContainerDied","Data":"5599e07e93bd0bd21a73574c52f271b8165b4822669f8ef25ea196f05151c565"}
Sep 08 16:46:17.527426 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:17.527300    2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5599e07e93bd0bd21a73574c52f271b8165b4822669f8ef25ea196f05151c565"
Sep 08 16:46:24.003659 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.003621    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"]
Sep 08 16:46:24.005727 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.005699    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-e22de8-29814765-xhjs2"]
Sep 08 16:46:24.032527 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.032496    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"]
Sep 08 16:46:24.032784 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.032755    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f" podUID="692cc1c2-4802-44c4-8444-1442a3a8e5ee" containerName="payload-processing" containerID="cri-o://4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0" gracePeriod=30
Sep 08 16:46:24.041942 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.041645    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"]
Sep 08 16:46:24.042181 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.042116    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp" podUID="96f84440-d65e-4968-84ed-c4360f5e468b" containerName="payload-pre-processing" containerID="cri-o://7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e" gracePeriod=30
Sep 08 16:46:24.208254 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.208219    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"]
Sep 08 16:46:24.209861 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.209836    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-e22de8-8t658"]
Sep 08 16:46:24.316773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.316750    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:46:24.320199 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.320180    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:46:24.440657 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.440622    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4g77\" (UniqueName: \"kubernetes.io/projected/96f84440-d65e-4968-84ed-c4360f5e468b-kube-api-access-n4g77\") pod \"96f84440-d65e-4968-84ed-c4360f5e468b\" (UID: \"96f84440-d65e-4968-84ed-c4360f5e468b\") "
Sep 08 16:46:24.440838 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.440709    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bgzn\" (UniqueName: \"kubernetes.io/projected/692cc1c2-4802-44c4-8444-1442a3a8e5ee-kube-api-access-7bgzn\") pod \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\" (UID: \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\") "
Sep 08 16:46:24.440838 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.440751    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f84440-d65e-4968-84ed-c4360f5e468b-plugins-config-volume\") pod \"96f84440-d65e-4968-84ed-c4360f5e468b\" (UID: \"96f84440-d65e-4968-84ed-c4360f5e468b\") "
Sep 08 16:46:24.440838 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.440775    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692cc1c2-4802-44c4-8444-1442a3a8e5ee-plugins-config-volume\") pod \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\" (UID: \"692cc1c2-4802-44c4-8444-1442a3a8e5ee\") "
Sep 08 16:46:24.441122 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.441100    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f84440-d65e-4968-84ed-c4360f5e468b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96f84440-d65e-4968-84ed-c4360f5e468b" (UID: "96f84440-d65e-4968-84ed-c4360f5e468b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:24.441190 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.441144    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692cc1c2-4802-44c4-8444-1442a3a8e5ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "692cc1c2-4802-44c4-8444-1442a3a8e5ee" (UID: "692cc1c2-4802-44c4-8444-1442a3a8e5ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 08 16:46:24.442773 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.442751    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f84440-d65e-4968-84ed-c4360f5e468b-kube-api-access-n4g77" (OuterVolumeSpecName: "kube-api-access-n4g77") pod "96f84440-d65e-4968-84ed-c4360f5e468b" (UID: "96f84440-d65e-4968-84ed-c4360f5e468b"). InnerVolumeSpecName "kube-api-access-n4g77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:24.442908 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.442889    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692cc1c2-4802-44c4-8444-1442a3a8e5ee-kube-api-access-7bgzn" (OuterVolumeSpecName: "kube-api-access-7bgzn") pod "692cc1c2-4802-44c4-8444-1442a3a8e5ee" (UID: "692cc1c2-4802-44c4-8444-1442a3a8e5ee"). InnerVolumeSpecName "kube-api-access-7bgzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:46:24.506878 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.506850    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="612c3829-11ce-4f72-8174-94eff2ec84c9" path="/var/lib/kubelet/pods/612c3829-11ce-4f72-8174-94eff2ec84c9/volumes"
Sep 08 16:46:24.507136 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.507124    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fcbe89f-ccc1-4751-a358-1a44be3db35a" path="/var/lib/kubelet/pods/7fcbe89f-ccc1-4751-a358-1a44be3db35a/volumes"
Sep 08 16:46:24.541889 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.541867    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4g77\" (UniqueName: \"kubernetes.io/projected/96f84440-d65e-4968-84ed-c4360f5e468b-kube-api-access-n4g77\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:46:24.541889 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.541889    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bgzn\" (UniqueName: \"kubernetes.io/projected/692cc1c2-4802-44c4-8444-1442a3a8e5ee-kube-api-access-7bgzn\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:46:24.542024 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.541898    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f84440-d65e-4968-84ed-c4360f5e468b-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:46:24.542024 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.541907    2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/692cc1c2-4802-44c4-8444-1442a3a8e5ee-plugins-config-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:46:24.562560 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.562494    2575 generic.go:356] "Generic (PLEG): container finished" podID="96f84440-d65e-4968-84ed-c4360f5e468b" containerID="7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e" exitCode=0
Sep 08 16:46:24.562560 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.562539    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"
Sep 08 16:46:24.562712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.562593    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp" event={"ID":"96f84440-d65e-4968-84ed-c4360f5e468b","Type":"ContainerDied","Data":"7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e"}
Sep 08 16:46:24.562712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.562646    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp" event={"ID":"96f84440-d65e-4968-84ed-c4360f5e468b","Type":"ContainerDied","Data":"d0fd09eefe54ccfa714b98c415ba0288561a0238b85c29929346c23e67bb153f"}
Sep 08 16:46:24.562712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.562674    2575 scope.go:117] "RemoveContainer" containerID="7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e"
Sep 08 16:46:24.564160 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.564106    2575 generic.go:356] "Generic (PLEG): container finished" podID="692cc1c2-4802-44c4-8444-1442a3a8e5ee" containerID="4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0" exitCode=0
Sep 08 16:46:24.564160 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.564153    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"
Sep 08 16:46:24.564331 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.564188    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f" event={"ID":"692cc1c2-4802-44c4-8444-1442a3a8e5ee","Type":"ContainerDied","Data":"4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0"}
Sep 08 16:46:24.564331 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.564226    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f" event={"ID":"692cc1c2-4802-44c4-8444-1442a3a8e5ee","Type":"ContainerDied","Data":"5f10fac1b4853e657185e6124e41cb35ded58532b839c9ea1745b7ab47018c75"}
Sep 08 16:46:24.571233 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.571214    2575 scope.go:117] "RemoveContainer" containerID="7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e"
Sep 08 16:46:24.571514 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:46:24.571495    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e\": container with ID starting with 7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e not found: ID does not exist" containerID="7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e"
Sep 08 16:46:24.571705 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.571525    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e"} err="failed to get container status \"7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e\": rpc error: code = NotFound desc = could not find container \"7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e\": container with ID starting with 7aeeb0d285d4ab60c089f2288aa70cf1e7091af49e6783a50e7556380ee84b4e not found: ID does not exist"
Sep 08 16:46:24.571705 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.571547    2575 scope.go:117] "RemoveContainer" containerID="4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0"
Sep 08 16:46:24.579371 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.579355    2575 scope.go:117] "RemoveContainer" containerID="4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0"
Sep 08 16:46:24.579651 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:46:24.579633    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0\": container with ID starting with 4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0 not found: ID does not exist" containerID="4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0"
Sep 08 16:46:24.579729 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.579655    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0"} err="failed to get container status \"4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0\": rpc error: code = NotFound desc = could not find container \"4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0\": container with ID starting with 4422dcdf96b8528f07fd07ca75b6922e86afd2748aadcac926da7fcbf4eba1f0 not found: ID does not exist"
Sep 08 16:46:24.580956 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.580937    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"]
Sep 08 16:46:24.586898 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.586879    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-e22de8-8c7757c74-7lfmp"]
Sep 08 16:46:24.598452 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.598426    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"]
Sep 08 16:46:24.599938 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:24.599919    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e22de8-b8c747ccc-67n4f"]
Sep 08 16:46:26.506818 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:26.506788    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="692cc1c2-4802-44c4-8444-1442a3a8e5ee" path="/var/lib/kubelet/pods/692cc1c2-4802-44c4-8444-1442a3a8e5ee/volumes"
Sep 08 16:46:26.507181 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:46:26.507120    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96f84440-d65e-4968-84ed-c4360f5e468b" path="/var/lib/kubelet/pods/96f84440-d65e-4968-84ed-c4360f5e468b/volumes"
Sep 08 16:47:46.603427 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:47:46.603397    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:47:46.608442 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:47:46.608417    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:47:46.609388 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:47:46.609363    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:47:46.614125 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:47:46.614106    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bbgqp_f317bd8d-0f4d-4df4-a6d1-fed8d9b54e4d/ovn-acl-logging/0.log"
Sep 08 16:47:47.525034 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:47:47.524996    2575 scope.go:117] "RemoveContainer" containerID="a15190a268fc939e0577e1df231008a6b719b40f999485c0861b7b43a8ba942d"
Sep 08 16:47:47.534655 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:47:47.534636    2575 scope.go:117] "RemoveContainer" containerID="0b08764330fab0a32df2c01da0d242de9bb008dcbdbcaf6c2b0c1cc428b513b8"
Sep 08 16:48:47.557922 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:47.557818    2575 scope.go:117] "RemoveContainer" containerID="505b3cb81e3349c8f4a2db2422c15990f2d254035f0d49e4be9288aabdfa98e3"
Sep 08 16:48:47.566089 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:47.566065    2575 scope.go:117] "RemoveContainer" containerID="cabf0664712a86bce9908182ae56d82149684a9bb2b35b97d12457b463f7bdcf"
Sep 08 16:48:54.750443 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:54.750410    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"]
Sep 08 16:48:54.750881 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:54.750639    2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv" podUID="45da06f9-39aa-44fd-bbcf-5e9fb10a298f" containerName="manager" containerID="cri-o://17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721" gracePeriod=10
Sep 08 16:48:56.606214 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:56.606191    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:48:56.669565 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:56.669538    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dksfx\" (UniqueName: \"kubernetes.io/projected/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-kube-api-access-dksfx\") pod \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\" (UID: \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\") "
Sep 08 16:48:56.669755 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:56.669609    2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-extensions-socket-volume\") pod \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\" (UID: \"45da06f9-39aa-44fd-bbcf-5e9fb10a298f\") "
Sep 08 16:48:56.669979 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:56.669954    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "45da06f9-39aa-44fd-bbcf-5e9fb10a298f" (UID: "45da06f9-39aa-44fd-bbcf-5e9fb10a298f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 08 16:48:56.671571 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:56.671541    2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-kube-api-access-dksfx" (OuterVolumeSpecName: "kube-api-access-dksfx") pod "45da06f9-39aa-44fd-bbcf-5e9fb10a298f" (UID: "45da06f9-39aa-44fd-bbcf-5e9fb10a298f"). InnerVolumeSpecName "kube-api-access-dksfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 08 16:48:56.770902 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:56.770876    2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dksfx\" (UniqueName: \"kubernetes.io/projected/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-kube-api-access-dksfx\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:48:56.770902 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:56.770899    2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/45da06f9-39aa-44fd-bbcf-5e9fb10a298f-extensions-socket-volume\") on node \"ip-10-0-128-92.ec2.internal\" DevicePath \"\""
Sep 08 16:48:57.356558 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.356525    2575 generic.go:356] "Generic (PLEG): container finished" podID="45da06f9-39aa-44fd-bbcf-5e9fb10a298f" containerID="17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721" exitCode=0
Sep 08 16:48:57.356742 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.356570    2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"
Sep 08 16:48:57.356742 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.356614    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv" event={"ID":"45da06f9-39aa-44fd-bbcf-5e9fb10a298f","Type":"ContainerDied","Data":"17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721"}
Sep 08 16:48:57.356742 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.356667    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv" event={"ID":"45da06f9-39aa-44fd-bbcf-5e9fb10a298f","Type":"ContainerDied","Data":"042c280f488755078f6a7cd28ca7f848c1ce2b5ac49e03f725c3d9ac2dca2e00"}
Sep 08 16:48:57.356742 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.356689    2575 scope.go:117] "RemoveContainer" containerID="17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721"
Sep 08 16:48:57.366404 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.366388    2575 scope.go:117] "RemoveContainer" containerID="17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721"
Sep 08 16:48:57.366673 ip-10-0-128-92 kubenswrapper[2575]: E0908 16:48:57.366652    2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721\": container with ID starting with 17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721 not found: ID does not exist" containerID="17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721"
Sep 08 16:48:57.366724 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.366682    2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721"} err="failed to get container status \"17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721\": rpc error: code = NotFound desc = could not find container \"17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721\": container with ID starting with 17f455bf7fbc0ce6571dcbe81d0457234697f60691ba9db1a9dd13bfc3df7721 not found: ID does not exist"
Sep 08 16:48:57.378714 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.378692    2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"]
Sep 08 16:48:57.382496 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:57.382478    2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-kfqxv"]
Sep 08 16:48:58.506712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:48:58.506674    2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45da06f9-39aa-44fd-bbcf-5e9fb10a298f" path="/var/lib/kubelet/pods/45da06f9-39aa-44fd-bbcf-5e9fb10a298f/volumes"
Sep 08 16:49:55.416630 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.416592    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"]
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.416964    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f84440-d65e-4968-84ed-c4360f5e468b" containerName="payload-pre-processing"
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.416975    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f84440-d65e-4968-84ed-c4360f5e468b" containerName="payload-pre-processing"
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.416987    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612c3829-11ce-4f72-8174-94eff2ec84c9" containerName="revoke-keys"
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.416993    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="612c3829-11ce-4f72-8174-94eff2ec84c9" containerName="revoke-keys"
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.417004    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="692cc1c2-4802-44c4-8444-1442a3a8e5ee" containerName="payload-processing"
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.417010    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="692cc1c2-4802-44c4-8444-1442a3a8e5ee" containerName="payload-processing"
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.417024    2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45da06f9-39aa-44fd-bbcf-5e9fb10a298f" containerName="manager"
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.417029    2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="45da06f9-39aa-44fd-bbcf-5e9fb10a298f" containerName="manager"
Sep 08 16:49:55.417082 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.417088    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="45da06f9-39aa-44fd-bbcf-5e9fb10a298f" containerName="manager"
Sep 08 16:49:55.417363 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.417095    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="612c3829-11ce-4f72-8174-94eff2ec84c9" containerName="revoke-keys"
Sep 08 16:49:55.417363 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.417103    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="692cc1c2-4802-44c4-8444-1442a3a8e5ee" containerName="payload-processing"
Sep 08 16:49:55.417363 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.417113    2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f84440-d65e-4968-84ed-c4360f5e468b" containerName="payload-pre-processing"
Sep 08 16:49:55.419938 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.419918    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:55.422515 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.422493    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-sm84v\""
Sep 08 16:49:55.434936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.434913    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"]
Sep 08 16:49:55.551441 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.551396    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tdmcg\" (UID: \"c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:55.551641 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.551496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lzgc\" (UniqueName: \"kubernetes.io/projected/c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72-kube-api-access-4lzgc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tdmcg\" (UID: \"c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:55.652215 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.652192    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tdmcg\" (UID: \"c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:55.652334 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.652228    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lzgc\" (UniqueName: \"kubernetes.io/projected/c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72-kube-api-access-4lzgc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tdmcg\" (UID: \"c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:55.652541 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.652525    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tdmcg\" (UID: \"c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:55.661410 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.661383    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lzgc\" (UniqueName: \"kubernetes.io/projected/c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72-kube-api-access-4lzgc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-tdmcg\" (UID: \"c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:55.730936 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:55.730875    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:56.023024 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:56.022982    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"]
Sep 08 16:49:56.656380 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:56.656345    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg" event={"ID":"c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72","Type":"ContainerStarted","Data":"2fccbb7a02fa99dd8278dcbd88cff0c38fac94eeea741887aabe152352c8bf08"}
Sep 08 16:49:56.656796 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:56.656396    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg" event={"ID":"c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72","Type":"ContainerStarted","Data":"b2eb570af8f67e80ca55a981db1f565c40b3720d9f68183ea0c5c4b82bb93ede"}
Sep 08 16:49:56.656796 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:56.656491    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:49:56.679667 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:49:56.679623    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg" podStartSLOduration=1.679610534 podStartE2EDuration="1.679610534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:49:56.678358207 +0000 UTC m=+2230.765021979" watchObservedRunningTime="2026-09-08 16:49:56.679610534 +0000 UTC m=+2230.766274282"
Sep 08 16:50:07.664121 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:50:07.664089    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-tdmcg"
Sep 08 16:50:47.602363 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:50:47.602330    2575 scope.go:117] "RemoveContainer" containerID="8a1c470d6a3860c64b0a6f19d8c833346aae7f223baf19d67e665a47cb3a2f65"
Sep 08 16:51:34.666429 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.666391    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"]
Sep 08 16:51:34.669792 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.669774    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.672717 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.672694    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ksssl\""
Sep 08 16:51:34.673815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.673787    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 08 16:51:34.673815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.673803    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 08 16:51:34.677441 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.677418    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ae8a17-e682-4809-916b-cfbd50360b43-openshift-service-ca\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.677562 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.677464    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ae8a17-e682-4809-916b-cfbd50360b43-maas-api-tls\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.677562 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.677496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ae8a17-e682-4809-916b-cfbd50360b43-metrics-certs\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.677742 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.677665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrt6\" (UniqueName: \"kubernetes.io/projected/54ae8a17-e682-4809-916b-cfbd50360b43-kube-api-access-dlrt6\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.680193 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.680174    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"]
Sep 08 16:51:34.778554 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.778515    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrt6\" (UniqueName: \"kubernetes.io/projected/54ae8a17-e682-4809-916b-cfbd50360b43-kube-api-access-dlrt6\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.778747 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.778590    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ae8a17-e682-4809-916b-cfbd50360b43-openshift-service-ca\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.778747 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.778622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ae8a17-e682-4809-916b-cfbd50360b43-maas-api-tls\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.778747 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.778649    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ae8a17-e682-4809-916b-cfbd50360b43-metrics-certs\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.779218 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.779195    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/54ae8a17-e682-4809-916b-cfbd50360b43-openshift-service-ca\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.781178 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.781152    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/54ae8a17-e682-4809-916b-cfbd50360b43-maas-api-tls\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.781263 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.781159    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/54ae8a17-e682-4809-916b-cfbd50360b43-metrics-certs\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.786359 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.786340    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrt6\" (UniqueName: \"kubernetes.io/projected/54ae8a17-e682-4809-916b-cfbd50360b43-kube-api-access-dlrt6\") pod \"maas-api-7bbd6f9566-xvqhk\" (UID: \"54ae8a17-e682-4809-916b-cfbd50360b43\") " pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:34.981072 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:34.981001    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:35.255191 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:35.255165    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"]
Sep 08 16:51:35.257124 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:51:35.257098    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ae8a17_e682_4809_916b_cfbd50360b43.slice/crio-34f10cd415c7eb334ef401e6f1101296534ec94d1cf85d1c3efef986a4efa1f1 WatchSource:0}: Error finding container 34f10cd415c7eb334ef401e6f1101296534ec94d1cf85d1c3efef986a4efa1f1: Status 404 returned error can't find the container with id 34f10cd415c7eb334ef401e6f1101296534ec94d1cf85d1c3efef986a4efa1f1
Sep 08 16:51:35.258345 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:35.258325    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 08 16:51:36.166712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:36.166673    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk" event={"ID":"54ae8a17-e682-4809-916b-cfbd50360b43","Type":"ContainerStarted","Data":"e06da18298de93a4cbbe61eb5acbac51d377f9ddb936e116957a458f6feef534"}
Sep 08 16:51:36.167120 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:36.166730    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk" event={"ID":"54ae8a17-e682-4809-916b-cfbd50360b43","Type":"ContainerStarted","Data":"34f10cd415c7eb334ef401e6f1101296534ec94d1cf85d1c3efef986a4efa1f1"}
Sep 08 16:51:36.167120 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:36.166833    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:36.183893 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:36.183831    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk" podStartSLOduration=1.735508062 podStartE2EDuration="2.183817259s" totalImagesPullingTime="448.309197ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-08 16:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:51:36.183660902 +0000 UTC m=+2330.270324652" watchObservedRunningTime="2026-09-08 16:51:36.183817259 +0000 UTC m=+2330.270481007"
Sep 08 16:51:42.177658 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:42.177565    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7bbd6f9566-xvqhk"
Sep 08 16:51:47.621855 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:47.621824    2575 scope.go:117] "RemoveContainer" containerID="06dad105d213fb99fa7ab9ac6434a3ceb82792eb9ad10ab56b25f0826f776759"
Sep 08 16:51:47.629933 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:51:47.629913    2575 scope.go:117] "RemoveContainer" containerID="ea5dbdffc5740b8bc3ef471c5a5fdc6dbb87a203d17630f694610ee514b29804"
Sep 08 16:52:01.276318 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:01.276287    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h9nxn_44279791-073f-4837-b065-4550a8ba5fca/manager/0.log"
Sep 08 16:52:01.858205 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:01.858171    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-jqcds_b77cf5b4-9e0c-4a24-a621-ce3396fb54cc/copy-manifests/0.log"
Sep 08 16:52:01.864951 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:01.864930    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-jqcds_b77cf5b4-9e0c-4a24-a621-ce3396fb54cc/manager/0.log"
Sep 08 16:52:02.425627 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:02.425597    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7bbd6f9566-xvqhk_54ae8a17-e682-4809-916b-cfbd50360b43/maas-api/0.log"
Sep 08 16:52:02.549815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:02.549785    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29814750-rq8jx_cd8c51df-0194-4b91-b0f3-3e521726bf87/cleanup/0.log"
Sep 08 16:52:02.771665 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:02.771571    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4z477_066d862e-2178-4d32-8644-f629c14f1e0e/postgres/0.log"
Sep 08 16:52:03.683463 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:03.683434    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-kj49v_fcc869a4-93d8-4ea3-ba93-ab7de1cd8797/manager/0.log"
Sep 08 16:52:03.800215 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:03.800185    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-w59v2_dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f/manager/0.log"
Sep 08 16:52:04.035044 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:04.034952    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-tdmcg_c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72/manager/0.log"
Sep 08 16:52:04.733265 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:04.733233    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pgh8w_4c88f4ae-d8bc-4be8-9228-d27b5ce35837/discovery/0.log"
Sep 08 16:52:04.844320 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:04.844287    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b99549648-ljldn_ef99e829-0519-4df3-b0a9-17c17d399e4c/kube-auth-proxy/0.log"
Sep 08 16:52:05.191757 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:05.191732    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9z79m_2a4a313a-9e47-4296-9b35-a07d2cc16aba/payload-pre-processing/0.log"
Sep 08 16:52:05.311332 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:05.311302    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xsx6m_71def932-1f2f-44ac-a578-0e90c4bfdf63/payload-processing/0.log"
Sep 08 16:52:05.419091 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:05.419065    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-849b99ddd-v5nhz_28efe48a-f884-4d00-a8dd-f33b1a14bae4/router/0.log"
Sep 08 16:52:05.758282 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:05.758247    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-6drxj_f01aea10-5231-4441-9136-b45cca535a4e/main/0.log"
Sep 08 16:52:06.188738 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:06.188708    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68frnl7j_713bc347-f48a-4330-8887-24d06cd4b18d/main/0.log"
Sep 08 16:52:06.295480 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:06.295448    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-4lm2n_4a2138c3-c9d7-41f6-a3ed-24eaa0e05ff6/main/0.log"
Sep 08 16:52:14.657343 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:14.657311    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fdnml_4133bb28-b1e2-4b78-9dc3-40043ee78b17/global-pull-secret-syncer/0.log"
Sep 08 16:52:14.831979 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:14.831944    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vd2fl_989fb239-f40a-405e-a570-8477f5c2a739/konnectivity-agent/0.log"
Sep 08 16:52:14.855452 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:14.855423    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-92.ec2.internal_b1bd51e38f398bda4d9557423e80c6a1/haproxy/0.log"
Sep 08 16:52:19.822949 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:19.822915    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-kj49v_fcc869a4-93d8-4ea3-ba93-ab7de1cd8797/manager/0.log"
Sep 08 16:52:19.846980 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:19.846954    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-w59v2_dfb0e707-dbd4-4aad-b7bc-6afb4ac8de7f/manager/0.log"
Sep 08 16:52:19.924368 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:19.924336    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-tdmcg_c2a00f17-4fb5-424d-9e8f-3fc31aaf3c72/manager/0.log"
Sep 08 16:52:21.912033 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:21.912005    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5b847f7b9-x4g9f_f4317f0d-665a-47ac-a7ea-fe020a0a0b40/metrics-server/0.log"
Sep 08 16:52:21.951815 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:21.951787    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-rv4f6_29221678-084d-4c62-98e3-2fe06530c52f/monitoring-plugin/0.log"
Sep 08 16:52:22.100637 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:22.100608    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9zx6_9a991395-90c0-4fd1-99e5-efa17b433253/node-exporter/0.log"
Sep 08 16:52:22.126221 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:22.126150    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9zx6_9a991395-90c0-4fd1-99e5-efa17b433253/kube-rbac-proxy/0.log"
Sep 08 16:52:22.149658 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:22.149641    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b9zx6_9a991395-90c0-4fd1-99e5-efa17b433253/init-textfile/0.log"
Sep 08 16:52:22.503414 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:22.503320    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-57bbw_520dadb4-b570-48b6-95b4-ef0f455fdfdc/prometheus-operator/0.log"
Sep 08 16:52:22.526173 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:22.526115    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-57bbw_520dadb4-b570-48b6-95b4-ef0f455fdfdc/kube-rbac-proxy/0.log"
Sep 08 16:52:22.592207 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:22.592175    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-mmjbq_707a51d2-c191-4bf0-8444-baa02cc338e7/prometheus-operator-admission-webhook/0.log"
Sep 08 16:52:24.449130 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.449096    2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"]
Sep 08 16:52:24.452442 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.452420    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.455003 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.454983    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfrvf\"/\"openshift-service-ca.crt\""
Sep 08 16:52:24.455959 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.455938    2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tfrvf\"/\"default-dockercfg-924bj\""
Sep 08 16:52:24.456048 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.455958    2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tfrvf\"/\"kube-root-ca.crt\""
Sep 08 16:52:24.460546 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.460524    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"]
Sep 08 16:52:24.576007 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.575977    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlkj9\" (UniqueName: \"kubernetes.io/projected/8c2901e6-205d-4fcd-8107-45ee47dfc330-kube-api-access-tlkj9\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.576184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.576019    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-sys\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.576184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.576049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-lib-modules\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.576184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.576100    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-proc\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.576184 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.576156    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-podres\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.676981 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.676950    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlkj9\" (UniqueName: \"kubernetes.io/projected/8c2901e6-205d-4fcd-8107-45ee47dfc330-kube-api-access-tlkj9\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.677159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.676992    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-sys\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.677159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.677024    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-lib-modules\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.677159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.677041    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-proc\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.677159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.677081    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-podres\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.677159 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.677094    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-sys\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.677344 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.677168    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-lib-modules\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.677344 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.677177    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-proc\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.677344 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.677196    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8c2901e6-205d-4fcd-8107-45ee47dfc330-podres\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.685712 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.685685    2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlkj9\" (UniqueName: \"kubernetes.io/projected/8c2901e6-205d-4fcd-8107-45ee47dfc330-kube-api-access-tlkj9\") pod \"perf-node-gather-daemonset-sjbx6\" (UID: \"8c2901e6-205d-4fcd-8107-45ee47dfc330\") " pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.763071 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.762991    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:24.832041 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.831995    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/2.log"
Sep 08 16:52:24.843268 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:24.843237    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-72szv_a4fce580-bb38-47e7-880f-90cafa36b3ec/console-operator/3.log"
Sep 08 16:52:25.031478 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:25.031454    2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"]
Sep 08 16:52:25.033922 ip-10-0-128-92 kubenswrapper[2575]: W0908 16:52:25.033899    2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8c2901e6_205d_4fcd_8107_45ee47dfc330.slice/crio-9508c6ca256b6cfdb01c9c369bfa65839ecbcd0e442aa71f1c90ce2dbb703b76 WatchSource:0}: Error finding container 9508c6ca256b6cfdb01c9c369bfa65839ecbcd0e442aa71f1c90ce2dbb703b76: Status 404 returned error can't find the container with id 9508c6ca256b6cfdb01c9c369bfa65839ecbcd0e442aa71f1c90ce2dbb703b76
Sep 08 16:52:25.377133 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:25.377104    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7f548ddff7-sshs6_62f3cb3c-19b9-4b57-b9fe-bed995fb98ff/console/0.log"
Sep 08 16:52:25.414800 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:25.414758    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6" event={"ID":"8c2901e6-205d-4fcd-8107-45ee47dfc330","Type":"ContainerStarted","Data":"9456c56a82c245a15518567cd6056dcf8d71079e07666827850c07a92a409256"}
Sep 08 16:52:25.414995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:25.414884    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:25.414995 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:25.414908    2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6" event={"ID":"8c2901e6-205d-4fcd-8107-45ee47dfc330","Type":"ContainerStarted","Data":"9508c6ca256b6cfdb01c9c369bfa65839ecbcd0e442aa71f1c90ce2dbb703b76"}
Sep 08 16:52:25.416567 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:25.416543    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-zdfj2_4b11db77-67a6-4083-8c4f-d701e15b92a8/download-server/0.log"
Sep 08 16:52:25.431698 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:25.431650    2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6" podStartSLOduration=1.43163752 podStartE2EDuration="1.43163752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-08 16:52:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-08 16:52:25.429988664 +0000 UTC m=+2379.516652413" watchObservedRunningTime="2026-09-08 16:52:25.43163752 +0000 UTC m=+2379.518301267"
Sep 08 16:52:27.244761 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:27.244730    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sl68l_ed691713-c786-4c14-b382-8bdedeb4073d/dns/0.log"
Sep 08 16:52:27.267698 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:27.267673    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sl68l_ed691713-c786-4c14-b382-8bdedeb4073d/kube-rbac-proxy/0.log"
Sep 08 16:52:27.290492 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:27.290473    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9l8gd_ef6c43f1-b4e3-464f-a4ab-57c86138262b/dns-node-resolver/0.log"
Sep 08 16:52:27.878739 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:27.878708    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dxwz6_6f8613b9-7b07-468f-a48c-ebb3d4d519e4/node-ca/0.log"
Sep 08 16:52:29.284709 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:29.284676    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pgh8w_4c88f4ae-d8bc-4be8-9228-d27b5ce35837/discovery/0.log"
Sep 08 16:52:29.315136 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:29.315099    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b99549648-ljldn_ef99e829-0519-4df3-b0a9-17c17d399e4c/kube-auth-proxy/0.log"
Sep 08 16:52:29.438910 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:29.438877    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9z79m_2a4a313a-9e47-4296-9b35-a07d2cc16aba/payload-pre-processing/0.log"
Sep 08 16:52:29.490687 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:29.490650    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xsx6m_71def932-1f2f-44ac-a578-0e90c4bfdf63/payload-processing/0.log"
Sep 08 16:52:29.517488 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:29.517453    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-849b99ddd-v5nhz_28efe48a-f884-4d00-a8dd-f33b1a14bae4/router/0.log"
Sep 08 16:52:30.384994 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:30.384963    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-fnfbq_d4599713-5ceb-4e41-a84e-0256209180b7/serve-healthcheck-canary/0.log"
Sep 08 16:52:31.038480 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:31.038452    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjvfx_ede55c97-e9a4-4599-95c5-9b35ca349745/kube-rbac-proxy/0.log"
Sep 08 16:52:31.059324 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:31.059299    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjvfx_ede55c97-e9a4-4599-95c5-9b35ca349745/exporter/0.log"
Sep 08 16:52:31.081715 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:31.081693    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rjvfx_ede55c97-e9a4-4599-95c5-9b35ca349745/extractor/0.log"
Sep 08 16:52:31.429804 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:31.429778    2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tfrvf/perf-node-gather-daemonset-sjbx6"
Sep 08 16:52:33.671596 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:33.671551    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-h9nxn_44279791-073f-4837-b065-4550a8ba5fca/manager/0.log"
Sep 08 16:52:34.031194 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:34.031164    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-jqcds_b77cf5b4-9e0c-4a24-a621-ce3396fb54cc/manager/0.log"
Sep 08 16:52:34.055404 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:34.055381    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-68b89c556d-jqcds_b77cf5b4-9e0c-4a24-a621-ce3396fb54cc/copy-manifests/0.log"
Sep 08 16:52:36.436224 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:36.436140    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pgh8w_4c88f4ae-d8bc-4be8-9228-d27b5ce35837/discovery/0.log"
Sep 08 16:52:36.447155 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:36.447130    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b99549648-ljldn_ef99e829-0519-4df3-b0a9-17c17d399e4c/kube-auth-proxy/0.log"
Sep 08 16:52:36.543314 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:36.543288    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9z79m_2a4a313a-9e47-4296-9b35-a07d2cc16aba/payload-pre-processing/0.log"
Sep 08 16:52:36.579744 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:36.579716    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xsx6m_71def932-1f2f-44ac-a578-0e90c4bfdf63/payload-processing/0.log"
Sep 08 16:52:36.595758 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:36.595736    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-849b99ddd-v5nhz_28efe48a-f884-4d00-a8dd-f33b1a14bae4/router/0.log"
Sep 08 16:52:38.460562 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:38.460529    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-pgh8w_4c88f4ae-d8bc-4be8-9228-d27b5ce35837/discovery/0.log"
Sep 08 16:52:38.476974 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:38.476936    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b99549648-ljldn_ef99e829-0519-4df3-b0a9-17c17d399e4c/kube-auth-proxy/0.log"
Sep 08 16:52:38.572716 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:38.572686    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-9z79m_2a4a313a-9e47-4296-9b35-a07d2cc16aba/payload-pre-processing/0.log"
Sep 08 16:52:38.610897 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:38.610867    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-xsx6m_71def932-1f2f-44ac-a578-0e90c4bfdf63/payload-processing/0.log"
Sep 08 16:52:38.629511 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:38.629482    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-849b99ddd-v5nhz_28efe48a-f884-4d00-a8dd-f33b1a14bae4/router/0.log"
Sep 08 16:52:42.231524 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:42.231496    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lp26m_0cab0104-2fcc-4735-a30b-b39095652998/kube-multus-additional-cni-plugins/0.log"
Sep 08 16:52:42.254330 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:42.254303    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lp26m_0cab0104-2fcc-4735-a30b-b39095652998/egress-router-binary-copy/0.log"
Sep 08 16:52:42.278475 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:42.278443    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lp26m_0cab0104-2fcc-4735-a30b-b39095652998/cni-plugins/0.log"
Sep 08 16:52:42.298448 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:42.298422    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lp26m_0cab0104-2fcc-4735-a30b-b39095652998/bond-cni-plugin/0.log"
Sep 08 16:52:42.318987 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:42.318952    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lp26m_0cab0104-2fcc-4735-a30b-b39095652998/routeoverride-cni/0.log"
Sep 08 16:52:42.338905 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:42.338874    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lp26m_0cab0104-2fcc-4735-a30b-b39095652998/whereabouts-cni-bincopy/0.log"
Sep 08 16:52:42.359355 ip-10-0-128-92 kubenswrapper[2575]: I0908 16:52:42.359328    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lp26m_0cab0104-2fcc-4735-a30b-b39095652998/whereabouts-cni/0.log"