Sep 11 19:24:18.048262 ip-10-0-137-97 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 19:24:18.048274 ip-10-0-137-97 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 19:24:18.048281 ip-10-0-137-97 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 19:24:18.048539 ip-10-0-137-97 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 19:24:27.889998 ip-10-0-137-97 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 19:24:27.890012 ip-10-0-137-97 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 35c200fbdbc1424ca01c7ee3eee835ed -- Sep 11 19:26:05.186425 ip-10-0-137-97 systemd[1]: Starting Kubernetes Kubelet... Sep 11 19:26:05.619834 ip-10-0-137-97 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 19:26:05.619834 ip-10-0-137-97 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 19:26:05.619834 ip-10-0-137-97 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 19:26:05.619834 ip-10-0-137-97 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 19:26:05.619834 ip-10-0-137-97 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 19:26:05.621664 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.621445 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 19:26:05.626933 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626908 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:26:05.626933 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626932 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:26:05.626933 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626936 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:26:05.626933 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626939 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:26:05.626933 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626941 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626945 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626947 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626950 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626953 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626956 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626958 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626961 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626963 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626966 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626969 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626971 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626974 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626976 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626979 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626981 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626984 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626987 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626990 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626992 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:26:05.627164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626995 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.626997 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627000 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627004 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627011 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627014 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627017 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627020 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627023 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627026 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627028 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627031 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627034 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627036 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627041 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627044 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627046 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627049 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627052 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:26:05.627741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627054 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627057 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627060 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627062 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627064 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627067 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627069 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627072 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627074 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627077 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627080 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627083 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627085 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627088 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627091 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627093 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627096 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627098 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627101 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627104 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:26:05.628215 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627106 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627108 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627111 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627113 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627116 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627120 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627139 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627146 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627150 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627153 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627156 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627159 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627162 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627164 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627167 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627170 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627172 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627176 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627179 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627182 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:26:05.628741 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627185 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627188 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627191 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627193 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627196 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627198 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627201 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627203 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627206 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627209 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627213 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627216 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627218 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627221 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627223 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627225 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627943 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627950 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627954 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:26:05.629259 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627957 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627960 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627963 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627966 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627968 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627971 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627974 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627976 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627979 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627981 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627984 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627986 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627989 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627991 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627994 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.627997 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628000 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628003 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628005 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628008 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:26:05.629719 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628010 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628013 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628015 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628018 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628020 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628023 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628025 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628029 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628033 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628036 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628039 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628042 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628046 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628048 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628051 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628053 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628056 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628059 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628061 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:26:05.630222 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628064 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628066 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628069 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628072 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628074 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628077 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628079 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628082 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628085 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628087 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628090 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628095 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628098 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628100 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628103 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628105 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628108 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628110 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628113 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628115 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:26:05.630718 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628117 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628120 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628137 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628140 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628142 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628148 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628151 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628153 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628156 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628159 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628162 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628164 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628167 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628170 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628172 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628175 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628177 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628179 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628182 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628184 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:26:05.631226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628187 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628191 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628194 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628196 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628199 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628202 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628205 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628208 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628210 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628213 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628215 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628218 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628221 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628223 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628227 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628231 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628234 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628629 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628634 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:26:05.631729 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628637 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628640 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628643 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628646 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628649 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628652 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628655 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628658 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628660 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628663 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628665 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628668 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628670 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628673 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628679 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628682 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628684 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628687 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628689 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628692 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:26:05.632205 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628695 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628698 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628700 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628703 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628705 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628709 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628712 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628714 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628717 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628719 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628723 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628725 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628728 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628731 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628733 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628736 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628739 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628741 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628744 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:26:05.632678 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628746 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628749 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628752 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628754 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628757 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628759 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628762 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628764 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628767 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628770 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628772 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628775 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628779 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628783 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628786 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628788 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628791 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628793 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628797 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:26:05.633144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628799 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628801 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628804 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628806 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628809 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628812 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628814 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628817 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628819 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628822 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628824 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628827 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628829 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628832 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628834 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628837 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628839 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628842 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628844 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628846 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:26:05.633642 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628850 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628853 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628855 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628858 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628860 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628863 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628865 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628867 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628870 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628872 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628876 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628879 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628882 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628885 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628888 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628891 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628894 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628896 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.628899 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.628982 2559 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 19:26:05.634162 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629017 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629023 2559 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629028 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629031 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629034 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629039 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629043 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629047 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629049 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629053 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629056 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629059 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629062 2559 flags.go:64] FLAG: --cgroup-root="" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629065 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629068 2559 flags.go:64] FLAG: --client-ca-file="" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629072 2559 flags.go:64] FLAG: --cloud-provider="external" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629075 2559 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629080 2559 flags.go:64] FLAG: --cluster-domain="" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629082 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629085 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629088 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629092 2559 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629096 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 19:26:05.634644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629099 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629102 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629105 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629108 2559 flags.go:64] FLAG: --contention-profiling="false" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629111 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629114 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629118 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629138 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629144 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629149 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629153 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629156 2559 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629159 2559 flags.go:64] FLAG: --enable-server="true" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629162 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629166 2559 flags.go:64] FLAG: --event-burst="100" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629169 2559 flags.go:64] FLAG: --event-qps="50" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629171 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629174 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629177 2559 flags.go:64] FLAG: --eviction-hard="" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629181 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629184 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629187 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629190 2559 flags.go:64] FLAG: --eviction-soft="" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629193 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629196 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 19:26:05.635201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629199 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629202 2559 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629205 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629207 2559 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629210 2559 flags.go:64] FLAG: --feature-gates="" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629214 2559 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629217 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629220 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629223 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629225 2559 flags.go:64] FLAG: --healthz-port="10248" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629228 2559 flags.go:64] FLAG: --help="false" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629231 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-137-97.ec2.internal" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629234 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629237 2559 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629240 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629243 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629247 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629249 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629252 2559 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629254 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629258 2559 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629260 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629264 2559 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629266 2559 flags.go:64] FLAG: --kube-reserved="" Sep 11 19:26:05.635822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629270 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629272 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629275 2559 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629278 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629281 2559 flags.go:64] FLAG: --lock-file="" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629285 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629288 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629291 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629296 2559 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629299 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629302 2559 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629305 2559 flags.go:64] FLAG: --logging-format="text" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629308 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629311 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629314 2559 flags.go:64] FLAG: --manifest-url="" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629316 2559 flags.go:64] FLAG: --manifest-url-header="" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629321 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629323 2559 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629327 2559 flags.go:64] FLAG: --max-pods="110" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629330 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629334 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629337 2559 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629340 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629343 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629346 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 19:26:05.636434 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629349 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629356 2559 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629359 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629362 2559 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629365 2559 flags.go:64] FLAG: --pod-cidr="" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629368 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629373 2559 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629376 2559 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629379 2559 flags.go:64] FLAG: --pods-per-core="0" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629381 2559 flags.go:64] FLAG: --port="10250" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629384 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629387 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01d8acb8526133982" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629390 2559 flags.go:64] FLAG: --qos-reserved="" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629394 2559 flags.go:64] FLAG: --read-only-port="10255" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629397 2559 flags.go:64] FLAG: --register-node="true" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629400 2559 flags.go:64] FLAG: --register-with-taints="" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629404 2559 flags.go:64] FLAG: --registry-burst="10" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629407 2559 flags.go:64] FLAG: --registry-qps="5" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629410 2559 flags.go:64] FLAG: --reserved-cpus="" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629413 2559 flags.go:64] FLAG: --reserved-memory="" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629416 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629419 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629422 2559 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629425 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629428 2559 flags.go:64] FLAG: --runonce="false" Sep 11 19:26:05.637071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629430 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629433 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629438 2559 flags.go:64] FLAG: --seccomp-default="false" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629440 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629443 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629446 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629449 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629452 2559 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629455 2559 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629458 2559 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629460 2559 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629463 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629466 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629469 2559 flags.go:64] FLAG: --system-cgroups="" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629471 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629477 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629479 2559 flags.go:64] FLAG: --tls-cert-file="" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629482 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629486 2559 flags.go:64] FLAG: --tls-min-version="" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629489 2559 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629493 2559 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629495 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629498 2559 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629501 2559 flags.go:64] FLAG: --v="2" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629505 2559 flags.go:64] FLAG: --version="false" Sep 11 19:26:05.637690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629509 2559 flags.go:64] FLAG: --vmodule="" Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629512 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.629515 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629629 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629632 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629636 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629639 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629642 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629645 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629649 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629652 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629655 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629658 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629660 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629663 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629665 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629668 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629671 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629673 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629676 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:26:05.638309 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629679 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629681 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629684 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629687 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629689 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629692 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629694 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629699 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629701 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629704 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629706 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629709 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629711 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629714 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629717 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629719 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629722 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629725 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629727 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629730 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:26:05.638806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629733 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629737 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629739 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629742 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629757 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629760 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629763 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629765 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629768 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629770 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629773 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629775 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629778 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629780 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629783 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629785 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629788 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629790 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629793 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629796 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:26:05.639321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629799 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629802 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629806 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629809 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629812 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629815 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629818 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629820 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629823 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629827 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629830 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629833 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629836 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629840 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629842 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629845 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629848 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629851 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629853 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:26:05.639821 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629856 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629858 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629861 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629864 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629867 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629869 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629872 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629874 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629877 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629879 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629882 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629884 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629888 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629891 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629893 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629896 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629899 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629901 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629904 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629906 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:26:05.640308 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629909 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629912 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.629916 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.630812 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.637769 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.637785 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.638556 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.638564 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638637 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638642 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638646 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638649 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638652 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638655 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:26:05.640804 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638658 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638663 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638667 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638670 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638673 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638677 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638681 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638684 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638686 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638689 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638692 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638694 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638697 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638700 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638702 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638705 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638707 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638710 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638713 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:26:05.641180 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638716 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638719 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638721 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638724 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638727 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638730 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638732 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638735 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638738 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638740 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638742 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638745 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638748 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638751 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638753 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638755 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638758 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638761 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638763 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638766 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:26:05.641633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638768 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638771 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638773 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638776 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638778 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638780 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638783 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638785 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638787 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638790 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638792 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638795 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638798 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638800 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638803 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638805 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638809 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638812 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638814 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638817 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:26:05.642114 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638819 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638823 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638827 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638830 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638832 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638835 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638838 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638841 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638843 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638846 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638848 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638851 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638853 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638856 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638858 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638861 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638863 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638866 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638868 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:26:05.642604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638870 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638873 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638875 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638878 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638880 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638882 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638885 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638888 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638890 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638894 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638896 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638899 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638901 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638904 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.638906 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:26:05.643063 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.638911 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639047 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639051 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639054 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639057 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639060 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639063 2559 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639066 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639068 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639071 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639073 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639076 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639078 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639081 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639083 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639086 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639088 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639090 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639094 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639096 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 19:26:05.643460 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639099 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639101 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639104 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639106 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639114 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639117 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639119 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639140 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639143 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639146 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639148 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639150 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639153 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639156 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639158 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639161 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639163 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639165 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639168 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639170 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 19:26:05.643965 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639173 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639175 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639177 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639180 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639182 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639185 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639187 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639190 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639192 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639195 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639198 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639200 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639203 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639205 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639208 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639210 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639213 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639221 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639224 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 19:26:05.644452 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639227 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639229 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639232 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639236 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639239 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639242 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639245 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639248 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639251 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639254 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639256 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639259 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639261 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639264 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639267 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639270 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639273 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639275 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 19:26:05.644908 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639279 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639282 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639285 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639287 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639290 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639292 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639295 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639298 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639300 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639302 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639304 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639313 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639315 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639323 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639326 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639329 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639332 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639334 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639337 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639340 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 19:26:05.645362 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639342 2559 feature_gate.go:328] unrecognized feature gate: Example Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639344 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:05.639347 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.639352 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.639491 2559 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.641888 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.642971 2559 server.go:1019] "Starting client certificate rotation" Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.643079 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.644380 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 19:26:05.645865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.645040 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 19:26:05.646358 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.646331 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 19:26:05.657753 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.657734 2559 log.go:25] "Validated CRI v1 runtime API" Sep 11 19:26:05.661900 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.661882 2559 log.go:25] "Validated CRI v1 image API" Sep 11 19:26:05.662805 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.662780 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 19:26:05.667833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.667813 2559 fs.go:135] Filesystem UUIDs: map[0a79836a-1e11-48b1-a88e-e3dde999b48f:/dev/nvme0n1p4 45ea56de-d7eb-4670-a94c-3a931d2855bc:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 11 19:26:05.667901 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.667833 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 19:26:05.674499 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.674378 2559 manager.go:216] Machine: {Timestamp:2026-09-11 19:26:05.672395495 +0000 UTC m=+0.355215488 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3121729 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a688bf168123c890cc4eeaa25c67 SystemUUID:ec28a688-bf16-8123-c890-cc4eeaa25c67 BootID:35c200fb-dbc1-424c-a01c-7ee3eee835ed Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:de:56:eb:c8:5f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:de:56:eb:c8:5f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:c4:ad:a4:50:ef Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 19:26:05.674499 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.674492 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 19:26:05.674627 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.674615 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 19:26:05.678051 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.678023 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 19:26:05.678227 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.678054 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 19:26:05.678293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.678271 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 19:26:05.678780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.678769 2559 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 19:26:05.678780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.678781 2559 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 19:26:05.678844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.678789 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 19:26:05.679727 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.679717 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 19:26:05.680447 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.680435 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 19:26:05.684586 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.684575 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 11 19:26:05.684711 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.684702 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 19:26:05.684755 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.684732 2559 kubelet.go:486] "Attempting to sync node with API server" Sep 11 19:26:05.684755 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.684743 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 19:26:05.684828 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.684757 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 19:26:05.684828 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.684766 2559 kubelet.go:398] "Adding apiserver pod source" Sep 11 19:26:05.684828 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.684774 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 19:26:05.685205 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.685193 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 19:26:05.685254 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.685212 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 19:26:05.687117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.687102 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 19:26:05.691376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.691357 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 19:26:05.691488 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.691403 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 19:26:05.693161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693148 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 19:26:05.693196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693167 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 19:26:05.693196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693173 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 19:26:05.693196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693178 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 19:26:05.693196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693186 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 19:26:05.693196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693192 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 19:26:05.693328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693206 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 19:26:05.693328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693211 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 19:26:05.693328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693218 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 19:26:05.693328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693224 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 19:26:05.693328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693232 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 19:26:05.693328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.693237 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 19:26:05.694268 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.694253 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 19:26:05.694268 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.694268 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 19:26:05.697111 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.697086 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 19:26:05.697199 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.697149 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 19:26:05.697199 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.697156 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 19:26:05.698357 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.698343 2559 server.go:1268] "Started kubelet" Sep 11 19:26:05.698419 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.698391 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 19:26:05.698537 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.698485 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 19:26:05.698586 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.698557 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 19:26:05.699310 ip-10-0-137-97 systemd[1]: Started Kubernetes Kubelet. Sep 11 19:26:05.700295 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.700039 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 19:26:05.700777 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.700763 2559 server.go:310] "Adding debug handlers to kubelet server" Sep 11 19:26:05.703055 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.702113 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-97.ec2.internal.18d45ac7b1a528c2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-97.ec2.internal,UID:ip-10-0-137-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-97.ec2.internal,},FirstTimestamp:2026-09-11 19:26:05.698312386 +0000 UTC m=+0.381132379,LastTimestamp:2026-09-11 19:26:05.698312386 +0000 UTC m=+0.381132379,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-97.ec2.internal,}" Sep 11 19:26:05.703657 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.703640 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 19:26:05.705672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.705657 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 19:26:05.706205 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.706189 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 19:26:05.706316 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.706296 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:05.706897 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.706880 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 19:26:05.706897 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.706884 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 19:26:05.707045 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.706907 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 19:26:05.707045 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707025 2559 reconstruct.go:97] "Volume reconstruction finished" Sep 11 19:26:05.707045 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707033 2559 reconciler.go:29] "Reconciler: start to sync state" Sep 11 19:26:05.707403 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707388 2559 factory.go:153] Registering CRI-O factory Sep 11 19:26:05.707464 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707407 2559 factory.go:223] Registration of the crio container factory successfully Sep 11 19:26:05.707517 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707479 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 19:26:05.707517 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707488 2559 factory.go:55] Registering systemd factory Sep 11 19:26:05.707517 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707498 2559 factory.go:223] Registration of the systemd container factory successfully Sep 11 19:26:05.707517 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707514 2559 factory.go:103] Registering Raw factory Sep 11 19:26:05.707708 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.707530 2559 manager.go:1196] Started watching for new ooms in manager Sep 11 19:26:05.708474 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.708461 2559 manager.go:349] Starting recovery of all containers Sep 11 19:26:05.714031 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.714002 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 19:26:05.714152 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.714109 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 19:26:05.719088 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.719069 2559 manager.go:354] Recovery completed Sep 11 19:26:05.721049 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.721024 2559 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 11 19:26:05.724042 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.724028 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:26:05.725926 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.725909 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:26:05.726010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.725945 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:26:05.726010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.725962 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:26:05.726574 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.726561 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 19:26:05.726574 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.726573 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 19:26:05.726664 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.726588 2559 state_mem.go:36] "Initialized new in-memory state store" Sep 11 19:26:05.726664 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.726562 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:26:05.726965 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.726951 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cghrs" Sep 11 19:26:05.727428 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.727415 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:26:05.727505 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.727447 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:26:05.727505 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.727457 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:26:05.728147 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.728069 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-97.ec2.internal.18d45ac7b34a8795 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-97.ec2.internal,UID:ip-10-0-137-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-97.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-97.ec2.internal,},FirstTimestamp:2026-09-11 19:26:05.725927317 +0000 UTC m=+0.408747316,LastTimestamp:2026-09-11 19:26:05.725927317 +0000 UTC m=+0.408747316,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-97.ec2.internal,}" Sep 11 19:26:05.728747 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.728736 2559 policy_none.go:49] "None policy: Start" Sep 11 19:26:05.728782 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.728750 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 19:26:05.728782 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.728760 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 19:26:05.730690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.730674 2559 policy_none.go:47] "Start" Sep 11 19:26:05.734107 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.734089 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cghrs" Sep 11 19:26:05.737203 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.737078 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-97.ec2.internal.18d45ac7b34ae91f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-97.ec2.internal,UID:ip-10-0-137-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-97.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-97.ec2.internal,},FirstTimestamp:2026-09-11 19:26:05.725952287 +0000 UTC m=+0.408772281,LastTimestamp:2026-09-11 19:26:05.725952287 +0000 UTC m=+0.408772281,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-97.ec2.internal,}" Sep 11 19:26:05.766697 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.766682 2559 manager.go:337] "Starting Device Plugin manager" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.766725 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.766735 2559 server.go:87] "Starting device plugin registration server" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.767008 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.767023 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.767106 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.767200 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.767209 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.767724 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 19:26:05.789539 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.767768 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:05.805218 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.805175 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 19:26:05.806488 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.806473 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 19:26:05.806591 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.806529 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 19:26:05.806591 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.806559 2559 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 19:26:05.806676 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.806603 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 19:26:05.809307 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.809285 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:26:05.867713 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.867684 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:26:05.869007 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.868990 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:26:05.869083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.869021 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:26:05.869083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.869031 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:26:05.869083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.869056 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-97.ec2.internal" Sep 11 19:26:05.878996 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.878925 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-97.ec2.internal" Sep 11 19:26:05.878996 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.878953 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-97.ec2.internal\": node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:05.903641 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.903614 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:05.906715 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.906692 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal"] Sep 11 19:26:05.906764 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.906758 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:26:05.907218 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.907198 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44d3e253afc49fede2cde755e84d5885-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal\" (UID: \"44d3e253afc49fede2cde755e84d5885\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:05.907264 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.907236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44d3e253afc49fede2cde755e84d5885-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal\" (UID: \"44d3e253afc49fede2cde755e84d5885\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:05.907741 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.907724 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:26:05.907803 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.907754 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:26:05.907803 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.907764 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:26:05.909854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.909841 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:26:05.909972 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.909959 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:05.910011 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.909987 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:26:05.910565 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.910542 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:26:05.910565 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.910567 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:26:05.910715 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.910583 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:26:05.910715 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.910594 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:26:05.910715 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.910615 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:26:05.910715 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.910624 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:26:05.912535 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.912516 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" Sep 11 19:26:05.912625 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.912544 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:26:05.913380 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.913365 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:26:05.913455 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.913390 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:26:05.913455 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:05.913403 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:26:05.940958 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.940936 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-97.ec2.internal\" not found" node="ip-10-0-137-97.ec2.internal" Sep 11 19:26:05.945468 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:05.945453 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-97.ec2.internal\" not found" node="ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.004749 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.004722 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.008005 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.007973 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44d3e253afc49fede2cde755e84d5885-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal\" (UID: \"44d3e253afc49fede2cde755e84d5885\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.008151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.008013 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/44d3e253afc49fede2cde755e84d5885-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal\" (UID: \"44d3e253afc49fede2cde755e84d5885\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.008151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.008039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44d3e253afc49fede2cde755e84d5885-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal\" (UID: \"44d3e253afc49fede2cde755e84d5885\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.008151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.008005 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44d3e253afc49fede2cde755e84d5885-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal\" (UID: \"44d3e253afc49fede2cde755e84d5885\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.105221 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.105191 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.108408 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.108392 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a03fbe7e7fc68e01ff8c561ad56ba60-config\") pod \"kube-apiserver-proxy-ip-10-0-137-97.ec2.internal\" (UID: \"9a03fbe7e7fc68e01ff8c561ad56ba60\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.205900 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.205825 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.209103 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.209079 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a03fbe7e7fc68e01ff8c561ad56ba60-config\") pod \"kube-apiserver-proxy-ip-10-0-137-97.ec2.internal\" (UID: \"9a03fbe7e7fc68e01ff8c561ad56ba60\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.209171 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.209135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a03fbe7e7fc68e01ff8c561ad56ba60-config\") pod \"kube-apiserver-proxy-ip-10-0-137-97.ec2.internal\" (UID: \"9a03fbe7e7fc68e01ff8c561ad56ba60\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.244245 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.244214 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.248941 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.248618 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" Sep 11 19:26:06.306526 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.306484 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.406926 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.406895 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.507326 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.507260 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.607745 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.607703 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.643025 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.643004 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 19:26:06.643428 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.643183 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 19:26:06.708110 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.708081 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.737734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.737700 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 19:21:07 +0000 UTC" deadline="2028-05-26 04:29:52.208950472 +0000 UTC" Sep 11 19:26:06.737734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.737729 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14937h3m45.471224216s" Sep 11 19:26:06.802075 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.802014 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 19:26:06.803183 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.803167 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientMemory" Sep 11 19:26:06.803264 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.803199 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 19:26:06.803264 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.803216 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeHasSufficientPID" Sep 11 19:26:06.808192 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.808172 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:06.846447 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:06.846230 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a03fbe7e7fc68e01ff8c561ad56ba60.slice/crio-050a37c7e6875e4a00a6286ae2ecc45e67f23b5513a8003196fe5ab736466e82 WatchSource:0}: Error finding container 050a37c7e6875e4a00a6286ae2ecc45e67f23b5513a8003196fe5ab736466e82: Status 404 returned error can't find the container with id 050a37c7e6875e4a00a6286ae2ecc45e67f23b5513a8003196fe5ab736466e82 Sep 11 19:26:06.846681 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:06.846656 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d3e253afc49fede2cde755e84d5885.slice/crio-72c37353a8033fb47d8f02d98236dc92c6f01e5a50e35d19e0bfa2d7e9e1c17d WatchSource:0}: Error finding container 72c37353a8033fb47d8f02d98236dc92c6f01e5a50e35d19e0bfa2d7e9e1c17d: Status 404 returned error can't find the container with id 72c37353a8033fb47d8f02d98236dc92c6f01e5a50e35d19e0bfa2d7e9e1c17d Sep 11 19:26:06.850726 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.850711 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:26:06.857681 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:06.857663 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:26:06.908381 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:06.908353 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-97.ec2.internal\" not found" Sep 11 19:26:07.004685 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:07.004657 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:26:07.006621 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:07.006607 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" Sep 11 19:26:07.019237 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:07.019218 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 19:26:07.020667 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:07.020655 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" Sep 11 19:26:07.030758 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:07.030740 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 19:26:07.031749 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:07.031736 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:26:09.012424 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.012391 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 19:26:09.096212 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.096181 2559 apiserver.go:52] "Watching apiserver" Sep 11 19:26:09.104156 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.103983 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 19:26:09.104458 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.104413 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-qtsgj","kube-system/konnectivity-agent-zv4k9","kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4","openshift-dns/node-resolver-48p69","openshift-multus/multus-additional-cni-plugins-ldhgs","openshift-multus/multus-ltpvh","openshift-network-diagnostics/network-check-target-fjxvk","openshift-network-operator/iptables-alerter-5lcfg","kube-system/global-pull-secret-syncer-dbf4v","openshift-cluster-node-tuning-operator/tuned-xnljh","openshift-image-registry/node-ca-vqrbw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal","openshift-multus/network-metrics-daemon-crcsn"] Sep 11 19:26:09.107454 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.107418 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.110317 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.110293 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 19:26:09.110800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.110555 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 19:26:09.110800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.110591 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 19:26:09.110800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.110563 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vfkst\"" Sep 11 19:26:09.110800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.110559 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 19:26:09.111055 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.110835 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.112940 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.112919 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:09.113036 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.112998 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:09.113477 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.113460 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 19:26:09.113721 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.113701 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7kkw4\"" Sep 11 19:26:09.113897 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.113882 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 19:26:09.115856 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.115379 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.116426 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.116406 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 19:26:09.117347 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.117325 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.118161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.117903 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 19:26:09.118161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.117962 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 19:26:09.118161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.118126 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 19:26:09.118377 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.118356 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8q9lb\"" Sep 11 19:26:09.120015 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.119996 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 19:26:09.120238 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.120222 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5cl97\"" Sep 11 19:26:09.120380 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.120366 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 19:26:09.120653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.120632 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.122009 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.121989 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.123232 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.123207 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 19:26:09.123369 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.123350 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mtlf7\"" Sep 11 19:26:09.123709 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.123688 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 19:26:09.124912 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.124893 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 19:26:09.125117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.125099 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 19:26:09.125233 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.125214 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rm699\"" Sep 11 19:26:09.125359 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.125303 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 19:26:09.126147 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.126129 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 19:26:09.126542 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.126527 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:09.126633 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.126609 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:09.126769 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.126749 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.126834 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.126801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-k8s-cni-cncf-io\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.126834 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.126830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-hostroot\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.126933 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.126851 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7nv2\" (UniqueName: \"kubernetes.io/projected/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-kube-api-access-c7nv2\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.127530 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-tuned\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.127614 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127595 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-tmp-dir\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.127671 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127639 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqmnl\" (UniqueName: \"kubernetes.io/projected/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-kube-api-access-dqmnl\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.127725 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127669 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-system-cni-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.127725 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127694 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-cni-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.127826 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127772 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-cni-bin\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.127911 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127834 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-modprobe-d\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.128000 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127936 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-var-lib-kubelet\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.128999 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.127992 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sl2g\" (UniqueName: \"kubernetes.io/projected/ec735588-5813-43d5-882c-1c84d0ffa19b-kube-api-access-6sl2g\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129089 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-device-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.129089 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7svgd\" (UniqueName: \"kubernetes.io/projected/63b607dc-c2e6-493a-a808-3bed222e9ef9-kube-api-access-7svgd\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.129089 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129077 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-cni-multus\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129100 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-conf-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129123 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-etc-kubernetes\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:09.129189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.129189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129173 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-registration-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.129189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129187 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129201 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129216 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysctl-conf\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129250 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-kubelet\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129264 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-lib-modules\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-hosts-file\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129292 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgkvz\" (UniqueName: \"kubernetes.io/projected/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-kube-api-access-qgkvz\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129306 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cnibin\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129327 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.129348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129340 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-cnibin\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129353 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-cni-binary-copy\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129367 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-socket-dir-parent\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129383 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysconfig\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlgj\" (UniqueName: \"kubernetes.io/projected/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-kube-api-access-xdlgj\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-netns\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129426 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-daemon-config\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-multus-certs\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129518 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-kubernetes\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129546 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysctl-d\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129574 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-sys\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129610 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-host\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec735588-5813-43d5-882c-1c84d0ffa19b-tmp\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129702 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-sys-fs\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129724 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-systemd\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129757 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-run\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.129876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.129786 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-socket-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.131156 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.131139 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:09.133414 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.133395 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.133509 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.133496 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:09.135764 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.135577 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 19:26:09.135764 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.135624 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 19:26:09.135764 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.135705 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9kmr9\"" Sep 11 19:26:09.135928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.135863 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.138793 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.138174 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.138793 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.138220 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 19:26:09.138793 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.138418 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 19:26:09.138793 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.138648 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sc9kv\"" Sep 11 19:26:09.139574 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.139554 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 19:26:09.139644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.139589 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 19:26:09.139705 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.139679 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 19:26:09.139846 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.139830 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 19:26:09.140758 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.140739 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pfbwm\"" Sep 11 19:26:09.140833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.140795 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 19:26:09.140891 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.140801 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 19:26:09.140988 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.140971 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 19:26:09.152481 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.152461 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-bw6bh" Sep 11 19:26:09.158395 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.158375 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-bw6bh" Sep 11 19:26:09.218662 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.218639 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 19:26:09.222317 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.222265 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" event={"ID":"9a03fbe7e7fc68e01ff8c561ad56ba60","Type":"ContainerStarted","Data":"050a37c7e6875e4a00a6286ae2ecc45e67f23b5513a8003196fe5ab736466e82"} Sep 11 19:26:09.223427 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.223398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" event={"ID":"44d3e253afc49fede2cde755e84d5885","Type":"ContainerStarted","Data":"72c37353a8033fb47d8f02d98236dc92c6f01e5a50e35d19e0bfa2d7e9e1c17d"} Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230062 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230158 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-sys-fs\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230192 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-run\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-socket-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-sys-fs\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-socket-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230416 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-run\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230492 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7nv2\" (UniqueName: \"kubernetes.io/projected/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-kube-api-access-c7nv2\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-systemd-units\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-tmp-dir\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230608 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-system-cni-dir\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqmnl\" (UniqueName: \"kubernetes.io/projected/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-kube-api-access-dqmnl\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-system-cni-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230711 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-cni-bin\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.232966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovn-node-metrics-cert\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sl2g\" (UniqueName: \"kubernetes.io/projected/ec735588-5813-43d5-882c-1c84d0ffa19b-kube-api-access-6sl2g\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-cni-multus\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230875 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-conf-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230907 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/321c2253-fa90-47e2-84b8-10400adb7630-dbus\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-ovn\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230963 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-cni-netd\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.230997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231029 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231061 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231092 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2fqr\" (UniqueName: \"kubernetes.io/projected/67bd5666-da6f-4330-ba77-538d370e7c2d-kube-api-access-d2fqr\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231121 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67bd5666-da6f-4330-ba77-538d370e7c2d-host-slash\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c478c692-0fc0-4dd7-8e9a-4360e4c73643-agent-certs\") pod \"konnectivity-agent-zv4k9\" (UID: \"c478c692-0fc0-4dd7-8e9a-4360e4c73643\") " pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-conf-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c478c692-0fc0-4dd7-8e9a-4360e4c73643-konnectivity-ca\") pod \"konnectivity-agent-zv4k9\" (UID: \"c478c692-0fc0-4dd7-8e9a-4360e4c73643\") " pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231244 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-etc-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-tmp-dir\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.233863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-cni-bin\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-system-cni-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovnkube-config\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgkvz\" (UniqueName: \"kubernetes.io/projected/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-kube-api-access-qgkvz\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-etc-selinux\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-cni-multus\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231630 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cnibin\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-kubelet\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231738 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-var-lib-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231764 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67bd5666-da6f-4330-ba77-538d370e7c2d-iptables-alerter-script\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-kubernetes\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231866 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cnibin\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231915 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysctl-d\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231918 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231948 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-sys\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.231979 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.234325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232011 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-sys\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-systemd\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232150 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232196 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-systemd\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232276 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-k8s-cni-cncf-io\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232292 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-kubernetes\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232343 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-k8s-cni-cncf-io\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232398 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-hostroot\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysctl-d\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac400821-4acd-4ab5-8d9a-03ae365353ec-serviceca\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232509 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-hostroot\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-systemd\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232603 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-node-log\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232634 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-tuned\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232712 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-cni-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232743 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac400821-4acd-4ab5-8d9a-03ae365353ec-host\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232771 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-run-netns\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.235124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232797 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232849 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovnkube-script-lib\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-modprobe-d\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.232962 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-var-lib-kubelet\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-cni-dir\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233041 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-var-lib-kubelet\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-device-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-modprobe-d\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233098 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7svgd\" (UniqueName: \"kubernetes.io/projected/63b607dc-c2e6-493a-a808-3bed222e9ef9-kube-api-access-7svgd\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-device-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233158 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-etc-kubernetes\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233231 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-env-overrides\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233264 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-etc-kubernetes\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233319 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233346 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-registration-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/63b607dc-c2e6-493a-a808-3bed222e9ef9-registration-dir\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.235946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233405 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-log-socket\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.233486 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233487 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwlw\" (UniqueName: \"kubernetes.io/projected/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-kube-api-access-ngwlw\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233513 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysctl-conf\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.233544 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:09.733523888 +0000 UTC m=+3.006031088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-kubelet\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233629 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysctl-conf\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233655 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fdhb\" (UniqueName: \"kubernetes.io/projected/ac400821-4acd-4ab5-8d9a-03ae365353ec-kube-api-access-6fdhb\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.233685 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-var-lib-kubelet\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234432 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-lib-modules\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-hosts-file\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234552 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-cnibin\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234594 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-lib-modules\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-cni-binary-copy\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.236670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234624 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-socket-dir-parent\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-netns\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234686 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-netns\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234685 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-daemon-config\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234729 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysconfig\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlgj\" (UniqueName: \"kubernetes.io/projected/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-kube-api-access-xdlgj\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-multus-certs\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234805 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/321c2253-fa90-47e2-84b8-10400adb7630-kubelet-config\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234828 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-slash\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234850 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-cni-bin\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234874 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234897 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-host\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.234919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec735588-5813-43d5-882c-1c84d0ffa19b-tmp\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-hosts-file\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235082 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-cni-binary-copy\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235091 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-sysconfig\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.237279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235146 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-cni-binary-copy\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.238039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-host-run-multus-certs\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.238039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235149 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec735588-5813-43d5-882c-1c84d0ffa19b-host\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.238039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235189 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-cnibin\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.238039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235384 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-socket-dir-parent\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.238039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.235592 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-multus-daemon-config\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.238303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.238234 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec735588-5813-43d5-882c-1c84d0ffa19b-etc-tuned\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.238396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.238375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec735588-5813-43d5-882c-1c84d0ffa19b-tmp\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.239790 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.239764 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sl2g\" (UniqueName: \"kubernetes.io/projected/ec735588-5813-43d5-882c-1c84d0ffa19b-kube-api-access-6sl2g\") pod \"tuned-xnljh\" (UID: \"ec735588-5813-43d5-882c-1c84d0ffa19b\") " pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.241040 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.240458 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqmnl\" (UniqueName: \"kubernetes.io/projected/8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7-kube-api-access-dqmnl\") pod \"multus-additional-cni-plugins-ldhgs\" (UID: \"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7\") " pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.241040 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.240930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgkvz\" (UniqueName: \"kubernetes.io/projected/5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2-kube-api-access-qgkvz\") pod \"node-resolver-48p69\" (UID: \"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2\") " pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.241040 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.240993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7nv2\" (UniqueName: \"kubernetes.io/projected/f52a92a5-ae1d-4524-9ea7-c44f944a6cea-kube-api-access-c7nv2\") pod \"multus-ltpvh\" (UID: \"f52a92a5-ae1d-4524-9ea7-c44f944a6cea\") " pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.241887 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.241786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7svgd\" (UniqueName: \"kubernetes.io/projected/63b607dc-c2e6-493a-a808-3bed222e9ef9-kube-api-access-7svgd\") pod \"aws-ebs-csi-driver-node-sjsz4\" (UID: \"63b607dc-c2e6-493a-a808-3bed222e9ef9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.242539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.242515 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlgj\" (UniqueName: \"kubernetes.io/projected/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-kube-api-access-xdlgj\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:09.336180 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-env-overrides\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336180 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336148 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-log-socket\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336180 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwlw\" (UniqueName: \"kubernetes.io/projected/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-kube-api-access-ngwlw\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336196 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fdhb\" (UniqueName: \"kubernetes.io/projected/ac400821-4acd-4ab5-8d9a-03ae365353ec-kube-api-access-6fdhb\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.336468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336253 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/321c2253-fa90-47e2-84b8-10400adb7630-kubelet-config\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.336468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-log-socket\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336368 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336411 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-slash\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336415 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/321c2253-fa90-47e2-84b8-10400adb7630-kubelet-config\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.336468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336457 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-cni-bin\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336486 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-slash\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336487 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-cni-bin\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336543 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-systemd-units\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336604 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.336622 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336642 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-env-overrides\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovn-node-metrics-cert\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.336685 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret podName:321c2253-fa90-47e2-84b8-10400adb7630 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:09.836667071 +0000 UTC m=+3.109174253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret") pod "global-pull-secret-syncer-dbf4v" (UID: "321c2253-fa90-47e2-84b8-10400adb7630") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336708 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-systemd-units\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336750 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/321c2253-fa90-47e2-84b8-10400adb7630-dbus\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336766 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-ovn\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336779 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-cni-netd\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2fqr\" (UniqueName: \"kubernetes.io/projected/67bd5666-da6f-4330-ba77-538d370e7c2d-kube-api-access-d2fqr\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.336817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336824 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67bd5666-da6f-4330-ba77-538d370e7c2d-host-slash\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c478c692-0fc0-4dd7-8e9a-4360e4c73643-agent-certs\") pod \"konnectivity-agent-zv4k9\" (UID: \"c478c692-0fc0-4dd7-8e9a-4360e4c73643\") " pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336842 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-ovn\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c478c692-0fc0-4dd7-8e9a-4360e4c73643-konnectivity-ca\") pod \"konnectivity-agent-zv4k9\" (UID: \"c478c692-0fc0-4dd7-8e9a-4360e4c73643\") " pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336868 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-etc-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/321c2253-fa90-47e2-84b8-10400adb7630-dbus\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovnkube-config\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336912 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-kubelet\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336927 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-var-lib-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-cni-netd\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336941 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67bd5666-da6f-4330-ba77-538d370e7c2d-iptables-alerter-script\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336967 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac400821-4acd-4ab5-8d9a-03ae365353ec-serviceca\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336913 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/67bd5666-da6f-4330-ba77-538d370e7c2d-host-slash\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.336986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-systemd\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337019 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-kubelet\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337004 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-node-log\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-etc-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac400821-4acd-4ab5-8d9a-03ae365353ec-host\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.337533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337112 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-run-systemd\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337115 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-node-log\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337139 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ac400821-4acd-4ab5-8d9a-03ae365353ec-host\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337536 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-var-lib-openvswitch\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337549 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ac400821-4acd-4ab5-8d9a-03ae365353ec-serviceca\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337585 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovnkube-config\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-run-netns\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337626 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337638 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/67bd5666-da6f-4330-ba77-538d370e7c2d-iptables-alerter-script\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337671 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337681 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovnkube-script-lib\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337691 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-host-run-netns\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.337843 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c478c692-0fc0-4dd7-8e9a-4360e4c73643-konnectivity-ca\") pod \"konnectivity-agent-zv4k9\" (UID: \"c478c692-0fc0-4dd7-8e9a-4360e4c73643\") " pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:09.338194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.338162 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovnkube-script-lib\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.339180 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.339156 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-ovn-node-metrics-cert\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.340056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.340039 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c478c692-0fc0-4dd7-8e9a-4360e4c73643-agent-certs\") pod \"konnectivity-agent-zv4k9\" (UID: \"c478c692-0fc0-4dd7-8e9a-4360e4c73643\") " pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:09.342783 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.342763 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:26:09.342881 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.342786 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:26:09.342881 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.342798 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bdwd5 for pod openshift-network-diagnostics/network-check-target-fjxvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:09.342881 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.342846 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5 podName:471b0ac6-4043-47e2-a5ac-0f258aa1da75 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:09.842832641 +0000 UTC m=+3.115339819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-bdwd5" (UniqueName: "kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5") pod "network-check-target-fjxvk" (UID: "471b0ac6-4043-47e2-a5ac-0f258aa1da75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:09.345010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.344974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2fqr\" (UniqueName: \"kubernetes.io/projected/67bd5666-da6f-4330-ba77-538d370e7c2d-kube-api-access-d2fqr\") pod \"iptables-alerter-5lcfg\" (UID: \"67bd5666-da6f-4330-ba77-538d370e7c2d\") " pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.345701 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.345683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fdhb\" (UniqueName: \"kubernetes.io/projected/ac400821-4acd-4ab5-8d9a-03ae365353ec-kube-api-access-6fdhb\") pod \"node-ca-vqrbw\" (UID: \"ac400821-4acd-4ab5-8d9a-03ae365353ec\") " pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.346581 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.346560 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwlw\" (UniqueName: \"kubernetes.io/projected/28c1a168-932a-41f9-b0eb-ec9f6e891ec7-kube-api-access-ngwlw\") pod \"ovnkube-node-qtsgj\" (UID: \"28c1a168-932a-41f9-b0eb-ec9f6e891ec7\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.420338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.420303 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ltpvh" Sep 11 19:26:09.428359 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.428336 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xnljh" Sep 11 19:26:09.440246 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.440222 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" Sep 11 19:26:09.445973 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.445958 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-48p69" Sep 11 19:26:09.454010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.453993 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vqrbw" Sep 11 19:26:09.462878 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.462857 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" Sep 11 19:26:09.471609 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.471591 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5lcfg" Sep 11 19:26:09.478459 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.478427 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:09.483281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.483259 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:09.740585 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.740554 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:09.740728 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.740672 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:09.740768 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.740737 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:10.740723515 +0000 UTC m=+4.013230694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:09.841450 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.841393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:09.841624 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.841531 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:09.841624 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.841589 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret podName:321c2253-fa90-47e2-84b8-10400adb7630 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:10.841574718 +0000 UTC m=+4.114081900 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret") pod "global-pull-secret-syncer-dbf4v" (UID: "321c2253-fa90-47e2-84b8-10400adb7630") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:09.942608 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:09.942577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:09.942812 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.942741 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:26:09.942812 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.942764 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:26:09.942812 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.942774 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bdwd5 for pod openshift-network-diagnostics/network-check-target-fjxvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:09.942974 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:09.942876 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5 podName:471b0ac6-4043-47e2-a5ac-0f258aa1da75 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:10.94285285 +0000 UTC m=+4.215360032 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdwd5" (UniqueName: "kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5") pod "network-check-target-fjxvk" (UID: "471b0ac6-4043-47e2-a5ac-0f258aa1da75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:10.092487 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.092458 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c1a168_932a_41f9_b0eb_ec9f6e891ec7.slice/crio-c8b7aa6061941f0220494055da101ba800966f52d7c04d88d2b64343d5b0bb9e WatchSource:0}: Error finding container c8b7aa6061941f0220494055da101ba800966f52d7c04d88d2b64343d5b0bb9e: Status 404 returned error can't find the container with id c8b7aa6061941f0220494055da101ba800966f52d7c04d88d2b64343d5b0bb9e Sep 11 19:26:10.093910 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.093889 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac400821_4acd_4ab5_8d9a_03ae365353ec.slice/crio-f074dc882605d060cbe32f01a0abd4ed5d0e69dd67f427965b1ee3070983cbb2 WatchSource:0}: Error finding container f074dc882605d060cbe32f01a0abd4ed5d0e69dd67f427965b1ee3070983cbb2: Status 404 returned error can't find the container with id f074dc882605d060cbe32f01a0abd4ed5d0e69dd67f427965b1ee3070983cbb2 Sep 11 19:26:10.095512 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.094988 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f889a5c_b68b_4c33_ad71_2aaa6a7a6ad7.slice/crio-49476e78a0c4db326f65310f0518d8aaa259c54eb02bbd03735358dba5b4ae6c WatchSource:0}: Error finding container 49476e78a0c4db326f65310f0518d8aaa259c54eb02bbd03735358dba5b4ae6c: Status 404 returned error can't find the container with id 49476e78a0c4db326f65310f0518d8aaa259c54eb02bbd03735358dba5b4ae6c Sep 11 19:26:10.098059 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.098038 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b607dc_c2e6_493a_a808_3bed222e9ef9.slice/crio-785c37576d13167fce0318144a83a78b1b9aea35e8b4a4d5e0cfdb7e7714d9df WatchSource:0}: Error finding container 785c37576d13167fce0318144a83a78b1b9aea35e8b4a4d5e0cfdb7e7714d9df: Status 404 returned error can't find the container with id 785c37576d13167fce0318144a83a78b1b9aea35e8b4a4d5e0cfdb7e7714d9df Sep 11 19:26:10.098958 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.098938 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52a92a5_ae1d_4524_9ea7_c44f944a6cea.slice/crio-f2a55a30a70e52b1b61f6167a4fda75e11b74b9ef0b0cb2e1ad379a805d8339c WatchSource:0}: Error finding container f2a55a30a70e52b1b61f6167a4fda75e11b74b9ef0b0cb2e1ad379a805d8339c: Status 404 returned error can't find the container with id f2a55a30a70e52b1b61f6167a4fda75e11b74b9ef0b0cb2e1ad379a805d8339c Sep 11 19:26:10.101770 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.101738 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aed7f34_1818_4df8_95ec_bb9f5ce3c6e2.slice/crio-2f409442d6c3ab24bbcea181ab837446f5b792dfa700e56fb3d359f4f0186b96 WatchSource:0}: Error finding container 2f409442d6c3ab24bbcea181ab837446f5b792dfa700e56fb3d359f4f0186b96: Status 404 returned error can't find the container with id 2f409442d6c3ab24bbcea181ab837446f5b792dfa700e56fb3d359f4f0186b96 Sep 11 19:26:10.102367 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.102349 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bd5666_da6f_4330_ba77_538d370e7c2d.slice/crio-089c9643ee3e87f9608a30056faa9d586209351f6de62ceac68714057c2698e0 WatchSource:0}: Error finding container 089c9643ee3e87f9608a30056faa9d586209351f6de62ceac68714057c2698e0: Status 404 returned error can't find the container with id 089c9643ee3e87f9608a30056faa9d586209351f6de62ceac68714057c2698e0 Sep 11 19:26:10.103723 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.103691 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec735588_5813_43d5_882c_1c84d0ffa19b.slice/crio-5e010476fbdb209c7bf63b15de5e40ef0b704db173f70f163ebaa0de8f8ac6b1 WatchSource:0}: Error finding container 5e010476fbdb209c7bf63b15de5e40ef0b704db173f70f163ebaa0de8f8ac6b1: Status 404 returned error can't find the container with id 5e010476fbdb209c7bf63b15de5e40ef0b704db173f70f163ebaa0de8f8ac6b1 Sep 11 19:26:10.108321 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:10.108293 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc478c692_0fc0_4dd7_8e9a_4360e4c73643.slice/crio-6e59d8c99f238eace6bdde747f2c7379dacb984e5ca5685c1a5788043b07e3de WatchSource:0}: Error finding container 6e59d8c99f238eace6bdde747f2c7379dacb984e5ca5685c1a5788043b07e3de: Status 404 returned error can't find the container with id 6e59d8c99f238eace6bdde747f2c7379dacb984e5ca5685c1a5788043b07e3de Sep 11 19:26:10.161982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.161935 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:21:09 +0000 UTC" deadline="2028-02-26 04:33:59.460814236 +0000 UTC" Sep 11 19:26:10.161982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.161978 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12777h7m49.298838843s" Sep 11 19:26:10.227963 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.227926 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" event={"ID":"9a03fbe7e7fc68e01ff8c561ad56ba60","Type":"ContainerStarted","Data":"83b6ff429fc0e08a407ce1c0b8e41b19c7e30db8d65fa842afd3a7bdceb7a463"} Sep 11 19:26:10.229076 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.229045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xnljh" event={"ID":"ec735588-5813-43d5-882c-1c84d0ffa19b","Type":"ContainerStarted","Data":"5e010476fbdb209c7bf63b15de5e40ef0b704db173f70f163ebaa0de8f8ac6b1"} Sep 11 19:26:10.230314 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.230273 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ltpvh" event={"ID":"f52a92a5-ae1d-4524-9ea7-c44f944a6cea","Type":"ContainerStarted","Data":"f2a55a30a70e52b1b61f6167a4fda75e11b74b9ef0b0cb2e1ad379a805d8339c"} Sep 11 19:26:10.231499 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.231431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" event={"ID":"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7","Type":"ContainerStarted","Data":"49476e78a0c4db326f65310f0518d8aaa259c54eb02bbd03735358dba5b4ae6c"} Sep 11 19:26:10.232728 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.232698 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vqrbw" event={"ID":"ac400821-4acd-4ab5-8d9a-03ae365353ec","Type":"ContainerStarted","Data":"f074dc882605d060cbe32f01a0abd4ed5d0e69dd67f427965b1ee3070983cbb2"} Sep 11 19:26:10.233880 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.233858 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"c8b7aa6061941f0220494055da101ba800966f52d7c04d88d2b64343d5b0bb9e"} Sep 11 19:26:10.235158 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.235127 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zv4k9" event={"ID":"c478c692-0fc0-4dd7-8e9a-4360e4c73643","Type":"ContainerStarted","Data":"6e59d8c99f238eace6bdde747f2c7379dacb984e5ca5685c1a5788043b07e3de"} Sep 11 19:26:10.236519 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.236497 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5lcfg" event={"ID":"67bd5666-da6f-4330-ba77-538d370e7c2d","Type":"ContainerStarted","Data":"089c9643ee3e87f9608a30056faa9d586209351f6de62ceac68714057c2698e0"} Sep 11 19:26:10.237946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.237918 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-48p69" event={"ID":"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2","Type":"ContainerStarted","Data":"2f409442d6c3ab24bbcea181ab837446f5b792dfa700e56fb3d359f4f0186b96"} Sep 11 19:26:10.239419 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.239398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" event={"ID":"63b607dc-c2e6-493a-a808-3bed222e9ef9","Type":"ContainerStarted","Data":"785c37576d13167fce0318144a83a78b1b9aea35e8b4a4d5e0cfdb7e7714d9df"} Sep 11 19:26:10.245720 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.245671 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-97.ec2.internal" podStartSLOduration=2.245658338 podStartE2EDuration="2.245658338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:10.244867669 +0000 UTC m=+3.517374870" watchObservedRunningTime="2026-09-11 19:26:10.245658338 +0000 UTC m=+3.518165538" Sep 11 19:26:10.748879 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.748837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:10.749126 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:10.749038 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:10.749126 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:10.749103 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:12.749083989 +0000 UTC m=+6.021591170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:10.849721 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.849671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:10.849928 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:10.849902 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:10.849990 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:10.849985 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret podName:321c2253-fa90-47e2-84b8-10400adb7630 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:12.849965113 +0000 UTC m=+6.122472305 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret") pod "global-pull-secret-syncer-dbf4v" (UID: "321c2253-fa90-47e2-84b8-10400adb7630") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:10.950950 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:10.950862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:10.951117 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:10.951054 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:26:10.951117 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:10.951078 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:26:10.951117 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:10.951091 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bdwd5 for pod openshift-network-diagnostics/network-check-target-fjxvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:10.951272 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:10.951153 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5 podName:471b0ac6-4043-47e2-a5ac-0f258aa1da75 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:12.951132456 +0000 UTC m=+6.223639636 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdwd5" (UniqueName: "kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5") pod "network-check-target-fjxvk" (UID: "471b0ac6-4043-47e2-a5ac-0f258aa1da75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:11.162898 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:11.162852 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 19:21:09 +0000 UTC" deadline="2028-03-12 02:18:35.402511023 +0000 UTC" Sep 11 19:26:11.162898 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:11.162897 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13134h52m24.239618701s" Sep 11 19:26:11.220145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:11.220059 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:11.220306 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:11.220205 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:11.220637 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:11.220616 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:11.220737 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:11.220717 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:11.220828 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:11.220784 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:11.220882 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:11.220858 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:11.259617 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:11.259583 2559 generic.go:356] "Generic (PLEG): container finished" podID="44d3e253afc49fede2cde755e84d5885" containerID="89f57c52c28c953aa95d1581c9680c2bd14e6f30a33c437d850763d1d30f1ecb" exitCode=0 Sep 11 19:26:11.259789 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:11.259676 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" event={"ID":"44d3e253afc49fede2cde755e84d5885","Type":"ContainerDied","Data":"89f57c52c28c953aa95d1581c9680c2bd14e6f30a33c437d850763d1d30f1ecb"} Sep 11 19:26:12.283514 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:12.283223 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" event={"ID":"44d3e253afc49fede2cde755e84d5885","Type":"ContainerStarted","Data":"0b06a35f8a814b6ffa68c74b0035001a14b49dcf6c03a22680510a661b904308"} Sep 11 19:26:12.297409 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:12.297356 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-97.ec2.internal" podStartSLOduration=4.297335406 podStartE2EDuration="4.297335406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:12.297219237 +0000 UTC m=+5.569726438" watchObservedRunningTime="2026-09-11 19:26:12.297335406 +0000 UTC m=+5.569842610" Sep 11 19:26:12.768514 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:12.768471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:12.768686 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:12.768665 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:12.768745 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:12.768730 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:16.768710431 +0000 UTC m=+10.041217622 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:12.870913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:12.870286 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:12.870913 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:12.870507 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:12.870913 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:12.870576 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret podName:321c2253-fa90-47e2-84b8-10400adb7630 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:16.870557876 +0000 UTC m=+10.143065069 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret") pod "global-pull-secret-syncer-dbf4v" (UID: "321c2253-fa90-47e2-84b8-10400adb7630") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:12.971186 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:12.971145 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:12.971368 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:12.971332 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:26:12.971368 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:12.971352 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:26:12.971368 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:12.971365 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bdwd5 for pod openshift-network-diagnostics/network-check-target-fjxvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:12.971556 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:12.971428 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5 podName:471b0ac6-4043-47e2-a5ac-0f258aa1da75 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:16.971407673 +0000 UTC m=+10.243914876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdwd5" (UniqueName: "kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5") pod "network-check-target-fjxvk" (UID: "471b0ac6-4043-47e2-a5ac-0f258aa1da75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:13.220143 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:13.220066 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:13.220293 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:13.220204 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:13.220605 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:13.220583 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:13.220709 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:13.220679 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:13.220773 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:13.220736 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:13.220824 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:13.220803 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:15.217577 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:15.217536 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:15.218043 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:15.217661 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:15.218043 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:15.217954 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:15.218043 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:15.218022 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:15.218270 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:15.218124 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:15.218270 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:15.218178 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:16.802347 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:16.801724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:16.802347 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:16.801934 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:16.802347 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:16.801993 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:24.801975183 +0000 UTC m=+18.074482368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:16.902916 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:16.902874 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:16.903085 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:16.903061 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:16.903162 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:16.903143 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret podName:321c2253-fa90-47e2-84b8-10400adb7630 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:24.903121983 +0000 UTC m=+18.175629179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret") pod "global-pull-secret-syncer-dbf4v" (UID: "321c2253-fa90-47e2-84b8-10400adb7630") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:17.004616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:17.004033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:17.004616 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:17.004209 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:26:17.004616 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:17.004229 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:26:17.004616 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:17.004241 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bdwd5 for pod openshift-network-diagnostics/network-check-target-fjxvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:17.004616 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:17.004292 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5 podName:471b0ac6-4043-47e2-a5ac-0f258aa1da75 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:25.004278469 +0000 UTC m=+18.276785651 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdwd5" (UniqueName: "kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5") pod "network-check-target-fjxvk" (UID: "471b0ac6-4043-47e2-a5ac-0f258aa1da75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:17.220606 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:17.218534 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:17.220606 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:17.218557 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:17.220606 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:17.218579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:17.220606 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:17.218666 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:17.220606 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:17.219153 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:17.220606 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:17.219244 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:19.219010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:19.218541 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:19.219010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:19.218589 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:19.219010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:19.218557 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:19.219010 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:19.218677 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:19.219010 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:19.218784 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:19.219010 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:19.218902 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:21.217644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:21.217557 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:21.218078 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:21.217557 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:21.218078 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:21.217693 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:21.218078 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:21.217783 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:21.218078 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:21.217564 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:21.218078 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:21.217868 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:23.217609 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:23.217567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:23.218018 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:23.217566 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:23.218018 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:23.217693 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:23.218018 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:23.217580 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:23.218018 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:23.217793 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:23.218018 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:23.217862 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:24.861634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:24.861596 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:24.862088 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:24.861739 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:24.862088 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:24.861838 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:40.861816635 +0000 UTC m=+34.134323825 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:24.962490 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:24.962447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:24.962682 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:24.962562 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:24.962682 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:24.962642 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret podName:321c2253-fa90-47e2-84b8-10400adb7630 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:40.962622477 +0000 UTC m=+34.235129660 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret") pod "global-pull-secret-syncer-dbf4v" (UID: "321c2253-fa90-47e2-84b8-10400adb7630") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:25.062862 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:25.062825 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:25.063075 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:25.062952 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 19:26:25.063075 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:25.062974 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 19:26:25.063075 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:25.062986 2559 projected.go:196] Error preparing data for projected volume kube-api-access-bdwd5 for pod openshift-network-diagnostics/network-check-target-fjxvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:25.063075 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:25.063049 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5 podName:471b0ac6-4043-47e2-a5ac-0f258aa1da75 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:41.063029359 +0000 UTC m=+34.335536540 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-bdwd5" (UniqueName: "kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5") pod "network-check-target-fjxvk" (UID: "471b0ac6-4043-47e2-a5ac-0f258aa1da75") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 19:26:25.217211 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:25.217172 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:25.217375 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:25.217183 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:25.217375 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:25.217322 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:25.217496 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:25.217374 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:25.217553 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:25.217482 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:25.217596 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:25.217581 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:27.218334 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:27.218304 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:27.218786 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:27.218390 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:27.218786 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:27.218402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:27.218786 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:27.218428 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:27.218786 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:27.218516 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:27.218786 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:27.218583 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:29.217913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.217716 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:29.218757 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.217717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:29.218757 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:29.217997 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:29.218757 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.217759 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:29.218757 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:29.218058 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:29.218757 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:29.218111 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:29.321513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.321478 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xnljh" event={"ID":"ec735588-5813-43d5-882c-1c84d0ffa19b","Type":"ContainerStarted","Data":"b43ee01163095e132210f82092f190a46b76669f050f08276da759b67bb1f004"} Sep 11 19:26:29.323455 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.323415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ltpvh" event={"ID":"f52a92a5-ae1d-4524-9ea7-c44f944a6cea","Type":"ContainerStarted","Data":"9cf9436e6671cef22b97fc63edf2ae27e8b094673e3c21b41043a96f47ee5c6b"} Sep 11 19:26:29.325281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.325256 2559 generic.go:356] "Generic (PLEG): container finished" podID="8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7" containerID="10dbce388087d9b926a59fe779235a95474e82455664d4b0abc73fdd073eed8f" exitCode=0 Sep 11 19:26:29.325405 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.325340 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" event={"ID":"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7","Type":"ContainerDied","Data":"10dbce388087d9b926a59fe779235a95474e82455664d4b0abc73fdd073eed8f"} Sep 11 19:26:29.327508 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.327474 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vqrbw" event={"ID":"ac400821-4acd-4ab5-8d9a-03ae365353ec","Type":"ContainerStarted","Data":"7ec9d558d521decced1e6db5199f0d26f970335525baa0d69f11e9db9fc3c1cd"} Sep 11 19:26:29.331271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.331026 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"baf81d818fd2dbf6ad3ca9c9f50cddce25c81b54e74dad87f7a4189405a723f0"} Sep 11 19:26:29.331271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.331089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"2dcbc8273a7143be24b5ab483b1a04137120f34320df0b6e7b5b6b2062dc3c3b"} Sep 11 19:26:29.331271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.331125 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"0dd786b2ec99b750ade137205ba7b9cf099794b9c356fceebf7b2b0ec4ec681b"} Sep 11 19:26:29.331271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.331172 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"ba97c95178358ab64cb578c93feb34814ee1bc0839c58aa21a598994ce54d66a"} Sep 11 19:26:29.331271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.331204 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"70bbd870e481aae24c6851415609e19fee3cb1b299a8a37d7d42bf9a44b81f4d"} Sep 11 19:26:29.331271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.331231 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"024bb1c09a2863d70c26960bf59f0b04d761d4ac75cf359c80ba16dbfb6afe48"} Sep 11 19:26:29.332722 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.332677 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zv4k9" event={"ID":"c478c692-0fc0-4dd7-8e9a-4360e4c73643","Type":"ContainerStarted","Data":"c06fa78ea7af2fee809f47f27750aaa60b8ff737ad2f648487a80177eb8c7c6e"} Sep 11 19:26:29.334278 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.334257 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-48p69" event={"ID":"5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2","Type":"ContainerStarted","Data":"ec4e393376212eb34b35f8e9ea7ea3798cc034abe47a883c300e45d35e66906e"} Sep 11 19:26:29.335907 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.335881 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" event={"ID":"63b607dc-c2e6-493a-a808-3bed222e9ef9","Type":"ContainerStarted","Data":"530f4a538ec3cda067891da2129f769f3134897433b72b97dcd2c0f1dec0e73d"} Sep 11 19:26:29.337918 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.337874 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xnljh" podStartSLOduration=4.163524884 podStartE2EDuration="22.337858489s" totalImagesPullingTime="18.174333605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:29.337255914 +0000 UTC m=+22.609763114" watchObservedRunningTime="2026-09-11 19:26:29.337858489 +0000 UTC m=+22.610365688" Sep 11 19:26:29.353748 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.353681 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ltpvh" podStartSLOduration=3.969803534 podStartE2EDuration="22.353665226s" totalImagesPullingTime="18.383861692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:29.353533719 +0000 UTC m=+22.626040919" watchObservedRunningTime="2026-09-11 19:26:29.353665226 +0000 UTC m=+22.626172425" Sep 11 19:26:29.394919 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.394890 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:29.395469 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.395405 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zv4k9" podStartSLOduration=7.754599238 podStartE2EDuration="22.395388217s" totalImagesPullingTime="14.640788979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:29.395124359 +0000 UTC m=+22.667631581" watchObservedRunningTime="2026-09-11 19:26:29.395388217 +0000 UTC m=+22.667895418" Sep 11 19:26:29.395815 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.395794 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:29.409219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.409070 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-48p69" podStartSLOduration=4.280746709 podStartE2EDuration="22.409056095s" totalImagesPullingTime="18.128309386s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:29.40869686 +0000 UTC m=+22.681204059" watchObservedRunningTime="2026-09-11 19:26:29.409056095 +0000 UTC m=+22.681563294" Sep 11 19:26:29.430966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.430923 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vqrbw" podStartSLOduration=4.268347584 podStartE2EDuration="22.430909715s" totalImagesPullingTime="18.162562131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:29.430558668 +0000 UTC m=+22.703065868" watchObservedRunningTime="2026-09-11 19:26:29.430909715 +0000 UTC m=+22.703416915" Sep 11 19:26:29.851336 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:29.851314 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 19:26:30.191170 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:30.191056 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T19:26:29.851331868Z","UUID":"5a52a8b3-1a9e-4048-a606-d3ffdb4fe16c","Handler":null,"Name":"","Endpoint":""} Sep 11 19:26:30.192863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:30.192838 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 19:26:30.192995 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:30.192872 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 19:26:30.340535 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:30.340488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5lcfg" event={"ID":"67bd5666-da6f-4330-ba77-538d370e7c2d","Type":"ContainerStarted","Data":"8f579470e6e1a45aabee950a4fcabc8736869be28eee264241f9e70f3525a0f5"} Sep 11 19:26:30.342816 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:30.342763 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" event={"ID":"63b607dc-c2e6-493a-a808-3bed222e9ef9","Type":"ContainerStarted","Data":"3cb3f4c943e5afc0de0170a8ed234bf2d82b393604b22ec81fe49e8c7b987380"} Sep 11 19:26:30.354665 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:30.354618 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5lcfg" podStartSLOduration=5.172530015 podStartE2EDuration="23.354599281s" totalImagesPullingTime="18.182069266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:30.354420325 +0000 UTC m=+23.626927526" watchObservedRunningTime="2026-09-11 19:26:30.354599281 +0000 UTC m=+23.627106483" Sep 11 19:26:31.218606 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:31.218579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:31.218707 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:31.218579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:31.218707 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:31.218690 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:31.218829 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:31.218771 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:31.218890 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:31.218852 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:31.218971 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:31.218947 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:31.349672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:31.349592 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"47ce864bea44f115f9c24aed13c4a1faa411606c4e0f1b114243472b577d998e"} Sep 11 19:26:31.352012 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:31.351981 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" event={"ID":"63b607dc-c2e6-493a-a808-3bed222e9ef9","Type":"ContainerStarted","Data":"509232637b2d402ce5d32b5d9fa03c6073cf036c57dd22396f3394596a7df7e4"} Sep 11 19:26:31.352133 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:31.352037 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 19:26:31.394147 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:31.394092 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sjsz4" podStartSLOduration=3.761655248 podStartE2EDuration="24.394074836s" totalImagesPullingTime="20.632419588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:31.393404665 +0000 UTC m=+24.665911866" watchObservedRunningTime="2026-09-11 19:26:31.394074836 +0000 UTC m=+24.666582036" Sep 11 19:26:33.217511 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:33.217469 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:33.218023 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:33.217523 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:33.218023 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:33.217468 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:33.218023 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:33.217616 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:33.218023 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:33.217719 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:33.218023 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:33.217825 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:34.297527 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:34.297318 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:34.298216 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:34.297624 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 19:26:34.298216 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:34.297961 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zv4k9" Sep 11 19:26:34.360394 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:34.360362 2559 generic.go:356] "Generic (PLEG): container finished" podID="8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7" containerID="0b08fe91b758f14ecfa238aad3161d748c010f9766cca55e3f8a9d177ebb4fb4" exitCode=0 Sep 11 19:26:34.360570 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:34.360411 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" event={"ID":"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7","Type":"ContainerDied","Data":"0b08fe91b758f14ecfa238aad3161d748c010f9766cca55e3f8a9d177ebb4fb4"} Sep 11 19:26:34.364892 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:34.364866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" event={"ID":"28c1a168-932a-41f9-b0eb-ec9f6e891ec7","Type":"ContainerStarted","Data":"845cb55a842ec5a223e421099888ce84c8690ea75f7f890a58e10bb511d6118e"} Sep 11 19:26:34.411693 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:34.411646 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" podStartSLOduration=9.001662425 podStartE2EDuration="27.411629283s" totalImagesPullingTime="18.409966858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:34.411191393 +0000 UTC m=+27.683698595" watchObservedRunningTime="2026-09-11 19:26:34.411629283 +0000 UTC m=+27.684136482" Sep 11 19:26:35.218220 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.218130 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:35.218220 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.218144 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:35.218420 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:35.218237 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:35.218420 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:35.218336 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:35.218420 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.218384 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:35.218625 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:35.218478 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:35.369607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.369571 2559 generic.go:356] "Generic (PLEG): container finished" podID="8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7" containerID="3f2587276779c8feca844b8a3ab4c6cf584d84f3e75626f9643ac0fcf2b3992e" exitCode=0 Sep 11 19:26:35.370100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.369650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" event={"ID":"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7","Type":"ContainerDied","Data":"3f2587276779c8feca844b8a3ab4c6cf584d84f3e75626f9643ac0fcf2b3992e"} Sep 11 19:26:35.370100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.369747 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 19:26:35.370285 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.370249 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:35.370408 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.370324 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:35.386981 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.386959 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:35.388518 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.388497 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:35.674687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.674481 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dbf4v"] Sep 11 19:26:35.674810 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.674775 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:35.674873 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:35.674857 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:35.679541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.679506 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-crcsn"] Sep 11 19:26:35.679683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.679658 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:35.679827 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:35.679804 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:35.680410 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.680219 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fjxvk"] Sep 11 19:26:35.680410 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:35.680311 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:35.680410 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:35.680381 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:36.374709 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:36.374622 2559 generic.go:356] "Generic (PLEG): container finished" podID="8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7" containerID="cb42c8a596111c2962a49b6958dc78d3b3da61034cd549045c2bb32a5ea330c4" exitCode=0 Sep 11 19:26:36.375148 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:36.374705 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" event={"ID":"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7","Type":"ContainerDied","Data":"cb42c8a596111c2962a49b6958dc78d3b3da61034cd549045c2bb32a5ea330c4"} Sep 11 19:26:36.375148 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:36.374786 2559 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 11 19:26:37.224529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:37.224496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:37.224743 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:37.224535 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:37.224743 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:37.224500 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:37.224743 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:37.224597 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:37.224743 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:37.224729 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:37.224935 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:37.224844 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:37.234094 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:37.234075 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:26:38.391159 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:38.391121 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" podUID="28c1a168-932a-41f9-b0eb-ec9f6e891ec7" containerName="ovnkube-controller" probeResult="failure" output="" Sep 11 19:26:39.217198 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:39.217164 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:39.217340 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:39.217167 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:39.217340 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:39.217276 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:26:39.217464 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:39.217362 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dbf4v" podUID="321c2253-fa90-47e2-84b8-10400adb7630" Sep 11 19:26:39.217464 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:39.217167 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:39.217552 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:39.217481 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fjxvk" podUID="471b0ac6-4043-47e2-a5ac-0f258aa1da75" Sep 11 19:26:40.878295 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:40.878257 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:40.878741 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:40.878392 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:40.878741 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:40.878480 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:12.878458064 +0000 UTC m=+66.150965244 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 19:26:40.979290 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:40.979250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:40.979503 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:40.979390 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:40.979503 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:40.979475 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret podName:321c2253-fa90-47e2-84b8-10400adb7630 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:12.979461094 +0000 UTC m=+66.251968273 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret") pod "global-pull-secret-syncer-dbf4v" (UID: "321c2253-fa90-47e2-84b8-10400adb7630") : object "kube-system"/"original-pull-secret" not registered Sep 11 19:26:40.995830 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:40.995801 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-97.ec2.internal" event="NodeReady" Sep 11 19:26:40.995994 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:40.995959 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 19:26:41.037874 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.037838 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76bbf4cc86-47pdj"] Sep 11 19:26:41.055688 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.055659 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn"] Sep 11 19:26:41.056115 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.056088 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.060008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.059982 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p67zb\"" Sep 11 19:26:41.060337 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.060316 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 19:26:41.060337 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.060331 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 19:26:41.060539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.060426 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 19:26:41.066217 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.066196 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 19:26:41.070219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.069536 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn"] Sep 11 19:26:41.070219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.069567 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76bbf4cc86-47pdj"] Sep 11 19:26:41.070219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.069584 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qqtnn"] Sep 11 19:26:41.070528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.070507 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn" Sep 11 19:26:41.074574 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.074533 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 19:26:41.074574 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.074535 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 19:26:41.074733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.074544 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-xqnps\"" Sep 11 19:26:41.080460 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.080423 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:41.085257 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.085206 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdwd5\" (UniqueName: \"kubernetes.io/projected/471b0ac6-4043-47e2-a5ac-0f258aa1da75-kube-api-access-bdwd5\") pod \"network-check-target-fjxvk\" (UID: \"471b0ac6-4043-47e2-a5ac-0f258aa1da75\") " pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:41.085919 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.085896 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zkfwb"] Sep 11 19:26:41.086049 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.086031 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qqtnn" Sep 11 19:26:41.088650 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.088629 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 19:26:41.088939 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.088922 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-kxtl7\"" Sep 11 19:26:41.089040 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.088937 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 19:26:41.089170 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.089155 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 19:26:41.098522 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.098433 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qqtnn"] Sep 11 19:26:41.098522 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.098487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zkfwb"] Sep 11 19:26:41.098651 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.098544 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.101366 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.101346 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 19:26:41.101492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.101374 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 19:26:41.101784 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.101769 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x76hz\"" Sep 11 19:26:41.181131 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.181131 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181099 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqwx\" (UniqueName: \"kubernetes.io/projected/447b6ece-d0e6-40cc-811c-faed4cdb4871-kube-api-access-7qqwx\") pod \"ingress-canary-qqtnn\" (UID: \"447b6ece-d0e6-40cc-811c-faed4cdb4871\") " pod="openshift-ingress-canary/ingress-canary-qqtnn" Sep 11 19:26:41.181376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f12941-963e-4098-aeac-eb39ce1ea795-tmp-dir\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.181376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181183 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-certificates\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.181376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181201 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbn5d\" (UniqueName: \"kubernetes.io/projected/bc1aefd6-c66e-4bdc-aa55-adee3f093f86-kube-api-access-zbn5d\") pod \"network-check-source-6f8b768455-lf4dn\" (UID: \"bc1aefd6-c66e-4bdc-aa55-adee3f093f86\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn" Sep 11 19:26:41.181376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181221 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-bound-sa-token\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.181376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22cg\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-kube-api-access-k22cg\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.181376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181266 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447b6ece-d0e6-40cc-811c-faed4cdb4871-cert\") pod \"ingress-canary-qqtnn\" (UID: \"447b6ece-d0e6-40cc-811c-faed4cdb4871\") " pod="openshift-ingress-canary/ingress-canary-qqtnn" Sep 11 19:26:41.181376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181364 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-image-registry-private-configuration\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.181672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06f12941-963e-4098-aeac-eb39ce1ea795-config-volume\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.181672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181508 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc70d804-93b3-44c6-8102-6f325002eb8f-ca-trust-extracted\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.181672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-installation-pull-secrets\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.181672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181591 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt6kv\" (UniqueName: \"kubernetes.io/projected/06f12941-963e-4098-aeac-eb39ce1ea795-kube-api-access-bt6kv\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.181672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-trusted-ca\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.181672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.181670 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.202456 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.202397 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zv4k9_c478c692-0fc0-4dd7-8e9a-4360e4c73643/konnectivity-agent/0.log" limit=1024 Sep 11 19:26:41.221042 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.221017 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:26:41.221206 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.221133 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:26:41.221206 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.221177 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:41.224000 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.223978 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 19:26:41.224316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.224294 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fsc\"" Sep 11 19:26:41.224418 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.224317 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 19:26:41.224490 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.224472 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2f99f\"" Sep 11 19:26:41.247259 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.247237 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:41.282551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282519 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-bound-sa-token\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.282734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282558 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22cg\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-kube-api-access-k22cg\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.282734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282587 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447b6ece-d0e6-40cc-811c-faed4cdb4871-cert\") pod \"ingress-canary-qqtnn\" (UID: \"447b6ece-d0e6-40cc-811c-faed4cdb4871\") " pod="openshift-ingress-canary/ingress-canary-qqtnn" Sep 11 19:26:41.282734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282617 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-image-registry-private-configuration\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.282955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282786 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06f12941-963e-4098-aeac-eb39ce1ea795-config-volume\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.282955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc70d804-93b3-44c6-8102-6f325002eb8f-ca-trust-extracted\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.282955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-installation-pull-secrets\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.282955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt6kv\" (UniqueName: \"kubernetes.io/projected/06f12941-963e-4098-aeac-eb39ce1ea795-kube-api-access-bt6kv\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.283155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.282975 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-trusted-ca\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.283155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.283155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.283155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqwx\" (UniqueName: \"kubernetes.io/projected/447b6ece-d0e6-40cc-811c-faed4cdb4871-kube-api-access-7qqwx\") pod \"ingress-canary-qqtnn\" (UID: \"447b6ece-d0e6-40cc-811c-faed4cdb4871\") " pod="openshift-ingress-canary/ingress-canary-qqtnn" Sep 11 19:26:41.283155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f12941-963e-4098-aeac-eb39ce1ea795-tmp-dir\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.283155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283113 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-certificates\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.283155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283130 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbn5d\" (UniqueName: \"kubernetes.io/projected/bc1aefd6-c66e-4bdc-aa55-adee3f093f86-kube-api-access-zbn5d\") pod \"network-check-source-6f8b768455-lf4dn\" (UID: \"bc1aefd6-c66e-4bdc-aa55-adee3f093f86\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn" Sep 11 19:26:41.283484 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283254 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc70d804-93b3-44c6-8102-6f325002eb8f-ca-trust-extracted\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.283484 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283341 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/06f12941-963e-4098-aeac-eb39ce1ea795-config-volume\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.283484 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.283356 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:26:41.283484 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.283370 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:26:41.283484 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.283430 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:26:41.783412886 +0000 UTC m=+35.055920070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:26:41.283797 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.283732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f12941-963e-4098-aeac-eb39ce1ea795-tmp-dir\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.283797 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.283790 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:26:41.283982 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.283838 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:41.783823657 +0000 UTC m=+35.056330835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:26:41.284714 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.284417 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-certificates\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.284714 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.284589 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-trusted-ca\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.285831 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.285781 2559 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zv4k9_c478c692-0fc0-4dd7-8e9a-4360e4c73643/konnectivity-agent/0.log" limit=1024 Sep 11 19:26:41.286298 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.286276 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/447b6ece-d0e6-40cc-811c-faed4cdb4871-cert\") pod \"ingress-canary-qqtnn\" (UID: \"447b6ece-d0e6-40cc-811c-faed4cdb4871\") " pod="openshift-ingress-canary/ingress-canary-qqtnn" Sep 11 19:26:41.286879 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.286862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-installation-pull-secrets\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.287003 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.286983 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-image-registry-private-configuration\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.292541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.292497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbn5d\" (UniqueName: \"kubernetes.io/projected/bc1aefd6-c66e-4bdc-aa55-adee3f093f86-kube-api-access-zbn5d\") pod \"network-check-source-6f8b768455-lf4dn\" (UID: \"bc1aefd6-c66e-4bdc-aa55-adee3f093f86\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn" Sep 11 19:26:41.294021 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.294002 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqwx\" (UniqueName: \"kubernetes.io/projected/447b6ece-d0e6-40cc-811c-faed4cdb4871-kube-api-access-7qqwx\") pod \"ingress-canary-qqtnn\" (UID: \"447b6ece-d0e6-40cc-811c-faed4cdb4871\") " pod="openshift-ingress-canary/ingress-canary-qqtnn" Sep 11 19:26:41.294185 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.294139 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-bound-sa-token\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.294344 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.294314 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt6kv\" (UniqueName: \"kubernetes.io/projected/06f12941-963e-4098-aeac-eb39ce1ea795-kube-api-access-bt6kv\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.295810 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.295793 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22cg\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-kube-api-access-k22cg\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.382079 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.381888 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn" Sep 11 19:26:41.398248 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.398216 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qqtnn" Sep 11 19:26:41.787560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.787524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:41.787848 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:41.787571 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:41.787848 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.787688 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:26:41.787848 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.787691 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:26:41.787848 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.787783 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:42.787758063 +0000 UTC m=+36.060265261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:26:41.787848 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.787698 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:26:41.788103 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:41.787864 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:26:42.787843403 +0000 UTC m=+36.060350584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:26:42.617077 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:42.617010 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fjxvk"] Sep 11 19:26:42.621924 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:42.620881 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qqtnn"] Sep 11 19:26:42.621924 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:42.621345 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod471b0ac6_4043_47e2_a5ac_0f258aa1da75.slice/crio-0d387de83e1147f045b530cb120e908f7b3b9c54a5467f3bc47ca0cbaa01327f WatchSource:0}: Error finding container 0d387de83e1147f045b530cb120e908f7b3b9c54a5467f3bc47ca0cbaa01327f: Status 404 returned error can't find the container with id 0d387de83e1147f045b530cb120e908f7b3b9c54a5467f3bc47ca0cbaa01327f Sep 11 19:26:42.621924 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:42.621869 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn"] Sep 11 19:26:42.624833 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:26:42.624812 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1aefd6_c66e_4bdc_aa55_adee3f093f86.slice/crio-5d1efd0afbc656d0cc16eaa25cda2ea6bbfc2f4cd2f19b1809303da189233dd1 WatchSource:0}: Error finding container 5d1efd0afbc656d0cc16eaa25cda2ea6bbfc2f4cd2f19b1809303da189233dd1: Status 404 returned error can't find the container with id 5d1efd0afbc656d0cc16eaa25cda2ea6bbfc2f4cd2f19b1809303da189233dd1 Sep 11 19:26:42.796575 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:42.796549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:42.796688 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:42.796595 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:42.796755 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:42.796690 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:26:42.796755 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:42.796751 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:44.796735992 +0000 UTC m=+38.069243174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:26:42.796755 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:42.796752 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:26:42.796874 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:42.796766 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:26:42.796874 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:42.796812 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:26:44.796797704 +0000 UTC m=+38.069304895 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:26:43.396731 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:43.396697 2559 generic.go:356] "Generic (PLEG): container finished" podID="8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7" containerID="f088fc3790940b7c91d2d41b25b0dc041d2717b4e5fd4066980006fb392bd5af" exitCode=0 Sep 11 19:26:43.396911 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:43.396787 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" event={"ID":"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7","Type":"ContainerDied","Data":"f088fc3790940b7c91d2d41b25b0dc041d2717b4e5fd4066980006fb392bd5af"} Sep 11 19:26:43.399596 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:43.399519 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qqtnn" event={"ID":"447b6ece-d0e6-40cc-811c-faed4cdb4871","Type":"ContainerStarted","Data":"d76003dfe01afa7f0d6c8698383654f63682fca3d52764069c412e6b707c0e5e"} Sep 11 19:26:43.401253 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:43.401222 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fjxvk" event={"ID":"471b0ac6-4043-47e2-a5ac-0f258aa1da75","Type":"ContainerStarted","Data":"0d387de83e1147f045b530cb120e908f7b3b9c54a5467f3bc47ca0cbaa01327f"} Sep 11 19:26:43.403020 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:43.402990 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn" event={"ID":"bc1aefd6-c66e-4bdc-aa55-adee3f093f86","Type":"ContainerStarted","Data":"5d1efd0afbc656d0cc16eaa25cda2ea6bbfc2f4cd2f19b1809303da189233dd1"} Sep 11 19:26:44.416579 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:44.416540 2559 generic.go:356] "Generic (PLEG): container finished" podID="8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7" containerID="4ebde996aecbcc1e797278ef0997538742ebc9c0d85688407055dc28e2cf1df7" exitCode=0 Sep 11 19:26:44.417220 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:44.416613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" event={"ID":"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7","Type":"ContainerDied","Data":"4ebde996aecbcc1e797278ef0997538742ebc9c0d85688407055dc28e2cf1df7"} Sep 11 19:26:44.816268 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:44.815696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:44.816268 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:44.816031 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:44.816268 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:44.815970 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:26:44.816268 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:44.816125 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:48.816101427 +0000 UTC m=+42.088608628 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:26:44.816268 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:44.816171 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:26:44.816268 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:44.816187 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:26:44.816268 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:44.816230 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:26:48.816219166 +0000 UTC m=+42.088726344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:26:45.423079 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.423041 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" event={"ID":"8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7","Type":"ContainerStarted","Data":"a9dd3119891bbb113367122c222eb8fd03adc50b4a4e558ab92ac2c87f7bf758"} Sep 11 19:26:45.448014 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.447962 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ldhgs" podStartSLOduration=8.261924563 podStartE2EDuration="38.447942595s" totalImagesPullingTime="30.186018032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:45.445842364 +0000 UTC m=+38.718349564" watchObservedRunningTime="2026-09-11 19:26:45.447942595 +0000 UTC m=+38.720449794" Sep 11 19:26:45.505546 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.505506 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz"] Sep 11 19:26:45.519242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.519213 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz"] Sep 11 19:26:45.519389 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.519319 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:45.522376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.522214 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 19:26:45.522376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.522271 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 19:26:45.522376 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.522271 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qb6rt\"" Sep 11 19:26:45.622067 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.622028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3dbcdeec-40a0-4813-943f-172a623b04c5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:45.622067 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.622068 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:45.723405 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.723369 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3dbcdeec-40a0-4813-943f-172a623b04c5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:45.723591 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.723413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:45.723655 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:45.723613 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:26:45.723701 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:45.723684 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:46.223668071 +0000 UTC m=+39.496175251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:26:45.724149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:45.724127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/3dbcdeec-40a0-4813-943f-172a623b04c5-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:46.227495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:46.227454 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:46.227676 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:46.227565 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:26:46.227676 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:46.227648 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:47.227626396 +0000 UTC m=+40.500133575 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:26:47.234428 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:47.234362 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:47.234811 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:47.234499 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:26:47.234811 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:47.234554 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:49.234540154 +0000 UTC m=+42.507047336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:26:47.429867 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:47.429826 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fjxvk" event={"ID":"471b0ac6-4043-47e2-a5ac-0f258aa1da75","Type":"ContainerStarted","Data":"74d2b6df30cb7efe5545d7eb1cbbc4751e67169c9fcf139637b50b8be3c0dc42"} Sep 11 19:26:47.430047 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:47.429961 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:26:47.431498 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:47.431471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn" event={"ID":"bc1aefd6-c66e-4bdc-aa55-adee3f093f86","Type":"ContainerStarted","Data":"34253cf486bd925072d5d9053c62d7fe863b98d600bd4785b35709448542d0c2"} Sep 11 19:26:47.432917 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:47.432893 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qqtnn" event={"ID":"447b6ece-d0e6-40cc-811c-faed4cdb4871","Type":"ContainerStarted","Data":"714e0d54dc6e2142e7750e22e34a60331c5e4aa67fa61a5a19b0e97020af673a"} Sep 11 19:26:47.460924 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:47.460809 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fjxvk" podStartSLOduration=36.004861591 podStartE2EDuration="40.460794037s" totalImagesPullingTime="4.455932446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:47.459943185 +0000 UTC m=+40.732450387" watchObservedRunningTime="2026-09-11 19:26:47.460794037 +0000 UTC m=+40.733301291" Sep 11 19:26:47.476757 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:47.476710 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-lf4dn" podStartSLOduration=24.0121112 podStartE2EDuration="28.476696859s" totalImagesPullingTime="4.464585659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:47.476111894 +0000 UTC m=+40.748619096" watchObservedRunningTime="2026-09-11 19:26:47.476696859 +0000 UTC m=+40.749204095" Sep 11 19:26:47.494103 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:47.494054 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qqtnn" podStartSLOduration=2.040426645 podStartE2EDuration="6.494039884s" totalImagesPullingTime="4.453613239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:26:47.492758888 +0000 UTC m=+40.765266090" watchObservedRunningTime="2026-09-11 19:26:47.494039884 +0000 UTC m=+40.766547085" Sep 11 19:26:48.846503 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:48.846462 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:48.846503 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:48.846508 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:48.846950 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:48.846609 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:26:48.846950 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:48.846608 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:26:48.846950 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:48.846676 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:56.846661317 +0000 UTC m=+50.119168494 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:26:48.846950 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:48.846621 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:26:48.846950 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:48.846734 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:26:56.8467237 +0000 UTC m=+50.119230881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:26:49.249209 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:49.249181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:49.249368 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:49.249354 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:26:49.249432 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:49.249422 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:26:53.249406851 +0000 UTC m=+46.521914029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:26:53.280763 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:53.280721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:26:53.281136 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:53.280874 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:26:53.281136 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:53.280947 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:01.280931813 +0000 UTC m=+54.553438991 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:26:56.905458 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:56.905396 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:26:56.905458 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:26:56.905459 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:26:56.906008 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:56.905582 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:26:56.906008 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:56.905587 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:26:56.906008 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:56.905673 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:12.905651319 +0000 UTC m=+66.178158513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:26:56.906008 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:56.905593 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:26:56.906008 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:26:56.905745 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:27:12.905711846 +0000 UTC m=+66.178219041 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:27:00.390723 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.390687 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp"] Sep 11 19:27:00.392846 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.392831 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.395730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.395702 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 11 19:27:00.396983 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.396959 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 19:27:00.397289 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.397272 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 11 19:27:00.408397 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.408374 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 11 19:27:00.408504 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.408410 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 19:27:00.408605 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.408590 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 11 19:27:00.408856 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.408843 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 19:27:00.420938 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.420919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp"] Sep 11 19:27:00.431088 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.431067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.431194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.431097 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-ca\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.431194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.431117 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.431194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.431134 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-hub\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.431194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.431156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5pqp\" (UniqueName: \"kubernetes.io/projected/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-kube-api-access-p5pqp\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.431324 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.431230 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.532041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.532010 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.532041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.532045 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-ca\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.532266 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.532065 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.532266 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.532118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-hub\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.532266 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.532149 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5pqp\" (UniqueName: \"kubernetes.io/projected/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-kube-api-access-p5pqp\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.532266 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.532245 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.532895 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.532873 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-additional-service-ca\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.535639 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.535615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-ca\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.535849 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.535829 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.536015 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.535862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-hub\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.536015 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.535895 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.544812 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.544790 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5pqp\" (UniqueName: \"kubernetes.io/projected/363bfb12-8f9d-4a94-b51f-572c4f2ac4c9-kube-api-access-p5pqp\") pod \"cluster-proxy-proxy-agent-c4858f78f-q6mzp\" (UID: \"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.713745 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.713647 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:27:00.983285 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:00.983206 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp"] Sep 11 19:27:00.987164 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:27:00.987133 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod363bfb12_8f9d_4a94_b51f_572c4f2ac4c9.slice/crio-1b18c1a9f564e3d0f66a2950c8a2659c2cff4a1e8026cf4bfc0d5877fc334a03 WatchSource:0}: Error finding container 1b18c1a9f564e3d0f66a2950c8a2659c2cff4a1e8026cf4bfc0d5877fc334a03: Status 404 returned error can't find the container with id 1b18c1a9f564e3d0f66a2950c8a2659c2cff4a1e8026cf4bfc0d5877fc334a03 Sep 11 19:27:01.337351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:01.337323 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:27:01.337519 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:01.337497 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:27:01.337588 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:01.337579 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:17.337561956 +0000 UTC m=+70.610069134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:27:01.468588 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:01.468552 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" event={"ID":"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9","Type":"ContainerStarted","Data":"1b18c1a9f564e3d0f66a2950c8a2659c2cff4a1e8026cf4bfc0d5877fc334a03"} Sep 11 19:27:06.484020 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:06.483984 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" event={"ID":"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9","Type":"ContainerStarted","Data":"4b41b58c4a86704e3f87f418eaa9d942edc40ad8c2fcd9fac1fecc6ccf49ba53"} Sep 11 19:27:06.484020 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:06.484025 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" event={"ID":"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9","Type":"ContainerStarted","Data":"b6b6766ccd3b1465868ebc0cd2e9aca640398f33531ffb9c414e718e4d17674c"} Sep 11 19:27:06.484428 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:06.484039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" event={"ID":"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9","Type":"ContainerStarted","Data":"2321f9c2a1109defca5321ca7ec68743169cfc849e1e42972781242b02f92923"} Sep 11 19:27:06.504877 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:06.504805 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" podStartSLOduration=1.925244068 podStartE2EDuration="6.504785429s" totalImagesPullingTime="4.579541361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:06.502583397 +0000 UTC m=+59.775090599" watchObservedRunningTime="2026-09-11 19:27:06.504785429 +0000 UTC m=+59.777292627" Sep 11 19:27:08.390424 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:08.390388 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtsgj" Sep 11 19:27:12.930030 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:12.929996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:27:12.930030 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:12.930034 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:27:12.930573 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:12.930058 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:27:12.930573 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:12.930167 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:27:12.930573 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:12.930200 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:27:12.930573 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:12.930214 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:27:12.930573 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:12.930252 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:44.930232068 +0000 UTC m=+98.202739267 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:27:12.930573 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:12.930269 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:27:44.930261785 +0000 UTC m=+98.202768962 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:27:12.933203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:12.933182 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 19:27:12.940217 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:12.940203 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 19:27:12.940276 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:12.940264 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:28:16.94024958 +0000 UTC m=+130.212756757 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : secret "metrics-daemon-secret" not found Sep 11 19:27:13.031236 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:13.031197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:27:13.034105 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:13.034087 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 19:27:13.044598 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:13.044573 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/321c2253-fa90-47e2-84b8-10400adb7630-original-pull-secret\") pod \"global-pull-secret-syncer-dbf4v\" (UID: \"321c2253-fa90-47e2-84b8-10400adb7630\") " pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:27:13.341330 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:13.341291 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dbf4v" Sep 11 19:27:13.638141 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:13.638065 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dbf4v"] Sep 11 19:27:13.641610 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:27:13.641577 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod321c2253_fa90_47e2_84b8_10400adb7630.slice/crio-4c3c2ef0e8f79f18ae27a0eac393543968ca8f4fd96c31e125cc17944da49050 WatchSource:0}: Error finding container 4c3c2ef0e8f79f18ae27a0eac393543968ca8f4fd96c31e125cc17944da49050: Status 404 returned error can't find the container with id 4c3c2ef0e8f79f18ae27a0eac393543968ca8f4fd96c31e125cc17944da49050 Sep 11 19:27:14.513353 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:14.513320 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dbf4v" event={"ID":"321c2253-fa90-47e2-84b8-10400adb7630","Type":"ContainerStarted","Data":"4c3c2ef0e8f79f18ae27a0eac393543968ca8f4fd96c31e125cc17944da49050"} Sep 11 19:27:17.363903 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:17.363856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:27:17.364332 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:17.364053 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:27:17.364332 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:17.364139 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:27:49.364122131 +0000 UTC m=+102.636629312 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:27:18.438030 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:18.437994 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fjxvk" Sep 11 19:27:19.530056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:19.530019 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dbf4v" event={"ID":"321c2253-fa90-47e2-84b8-10400adb7630","Type":"ContainerStarted","Data":"8f919fd8f31832b333b3238421971b20df7982ab02b920ffee5d21f505bbfd74"} Sep 11 19:27:19.547960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:19.547910 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dbf4v" podStartSLOduration=67.63230141 podStartE2EDuration="1m12.547894485s" totalImagesPullingTime="4.915593075s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:27:19.546755245 +0000 UTC m=+72.819262445" watchObservedRunningTime="2026-09-11 19:27:19.547894485 +0000 UTC m=+72.820401682" Sep 11 19:27:44.972039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:44.971880 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:27:44.972039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:44.971937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:27:44.972039 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:44.972035 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:27:44.972039 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:44.972046 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:27:44.972648 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:44.972060 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:27:44.972648 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:44.972108 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:28:48.972094502 +0000 UTC m=+162.244601679 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:27:44.972648 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:44.972136 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:28:48.972116252 +0000 UTC m=+162.244623433 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:27:49.403993 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:27:49.403952 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:27:49.404367 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:49.404075 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:27:49.404367 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:27:49.404137 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:28:53.404123673 +0000 UTC m=+166.676630855 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:28:17.007391 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:28:17.007339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:28:17.007909 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:17.007508 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 19:28:17.007909 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:17.007582 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:19.007567308 +0000 UTC m=+252.280074486 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : secret "metrics-daemon-secret" not found Sep 11 19:28:44.068292 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:44.068233 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" podUID="cc70d804-93b3-44c6-8102-6f325002eb8f" Sep 11 19:28:44.106827 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:44.106790 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zkfwb" podUID="06f12941-963e-4098-aeac-eb39ce1ea795" Sep 11 19:28:44.232902 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:44.232844 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:28:44.772212 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:28:44.772186 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:28:44.772386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:28:44.772190 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zkfwb" Sep 11 19:28:48.530735 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:48.530689 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" podUID="3dbcdeec-40a0-4813-943f-172a623b04c5" Sep 11 19:28:48.782355 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:28:48.782276 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:28:49.041526 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:28:49.041403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:28:49.041526 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:28:49.041509 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:28:49.041717 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:49.041569 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 19:28:49.041717 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:49.041599 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 19:28:49.041717 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:49.041611 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76bbf4cc86-47pdj: secret "image-registry-tls" not found Sep 11 19:28:49.041717 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:49.041654 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls podName:06f12941-963e-4098-aeac-eb39ce1ea795 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:51.041635233 +0000 UTC m=+284.314142462 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls") pod "dns-default-zkfwb" (UID: "06f12941-963e-4098-aeac-eb39ce1ea795") : secret "dns-default-metrics-tls" not found Sep 11 19:28:49.041717 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:49.041669 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls podName:cc70d804-93b3-44c6-8102-6f325002eb8f nodeName:}" failed. No retries permitted until 2026-09-11 19:30:51.041663228 +0000 UTC m=+284.314170406 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls") pod "image-registry-76bbf4cc86-47pdj" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f") : secret "image-registry-tls" not found Sep 11 19:28:53.471748 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:28:53.471707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:28:53.472200 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:53.471869 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 11 19:28:53.472200 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:28:53.471934 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert podName:3dbcdeec-40a0-4813-943f-172a623b04c5 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:55.471918188 +0000 UTC m=+288.744425365 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-94njz" (UID: "3dbcdeec-40a0-4813-943f-172a623b04c5") : secret "networking-console-plugin-cert" not found Sep 11 19:28:56.217698 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:28:56.217652 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:30:19.095709 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:19.095664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:30:19.096216 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:19.095769 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 19:30:19.096216 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:19.095839 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs podName:0b2ec797-aa4b-4aa4-8472-782b11c2e9e4 nodeName:}" failed. No retries permitted until 2026-09-11 19:32:21.09582316 +0000 UTC m=+374.368330338 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs") pod "network-metrics-daemon-crcsn" (UID: "0b2ec797-aa4b-4aa4-8472-782b11c2e9e4") : secret "metrics-daemon-secret" not found Sep 11 19:30:34.045806 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.045728 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rd8p6"] Sep 11 19:30:34.048890 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.048869 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.051704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.051683 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 19:30:34.051897 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.051884 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 19:30:34.051959 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.051946 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 19:30:34.052005 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.051969 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 19:30:34.053105 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.053088 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-s6r98\"" Sep 11 19:30:34.065666 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.065645 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rd8p6"] Sep 11 19:30:34.099300 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.099269 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.099300 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.099300 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.099524 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.099395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/11495f40-d751-4bc7-8fc6-ed4d06014372-crio-socket\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.099524 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.099462 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/11495f40-d751-4bc7-8fc6-ed4d06014372-data-volume\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.099524 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.099480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7zvs\" (UniqueName: \"kubernetes.io/projected/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-api-access-d7zvs\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.200518 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.200476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.200684 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.200516 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.200684 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:34.200605 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:30:34.200684 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.200613 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/11495f40-d751-4bc7-8fc6-ed4d06014372-crio-socket\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.200684 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.200639 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/11495f40-d751-4bc7-8fc6-ed4d06014372-data-volume\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.200684 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.200660 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7zvs\" (UniqueName: \"kubernetes.io/projected/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-api-access-d7zvs\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.200684 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:34.200687 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:34.700666232 +0000 UTC m=+267.973173409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : configmap "kube-rbac-proxy" not found Sep 11 19:30:34.200961 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:34.200706 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:30:34.200961 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.200768 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/11495f40-d751-4bc7-8fc6-ed4d06014372-crio-socket\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.200961 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:34.200790 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:34.700771524 +0000 UTC m=+267.973278716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : secret "insights-runtime-extractor-tls" not found Sep 11 19:30:34.201116 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.201066 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/11495f40-d751-4bc7-8fc6-ed4d06014372-data-volume\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.209936 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.209901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7zvs\" (UniqueName: \"kubernetes.io/projected/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-api-access-d7zvs\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.704366 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.704329 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.704366 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:34.704371 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:34.704601 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:34.704494 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:30:34.704601 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:34.704557 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:30:34.704601 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:34.704590 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:35.704567365 +0000 UTC m=+268.977074546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : configmap "kube-rbac-proxy" not found Sep 11 19:30:34.704705 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:34.704609 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:35.70459828 +0000 UTC m=+268.977105483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : secret "insights-runtime-extractor-tls" not found Sep 11 19:30:35.710960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:35.710922 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:35.710960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:35.710963 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:35.711413 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:35.711039 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:30:35.711413 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:35.711059 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:30:35.711413 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:35.711103 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:37.71108668 +0000 UTC m=+270.983593880 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : configmap "kube-rbac-proxy" not found Sep 11 19:30:35.711413 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:35.711179 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:37.711162732 +0000 UTC m=+270.983669910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : secret "insights-runtime-extractor-tls" not found Sep 11 19:30:35.808608 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:35.808576 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-48p69_5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2/dns-node-resolver/0.log" Sep 11 19:30:37.213083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:37.213057 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vqrbw_ac400821-4acd-4ab5-8d9a-03ae365353ec/node-ca/0.log" Sep 11 19:30:37.726346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:37.726298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:37.726346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:37.726343 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:37.726584 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:37.726425 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 19:30:37.726584 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:37.726480 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:30:37.726584 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:37.726518 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:41.726500507 +0000 UTC m=+274.999007685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : configmap "kube-rbac-proxy" not found Sep 11 19:30:37.726584 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:37.726536 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:41.726524893 +0000 UTC m=+274.999032071 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : secret "insights-runtime-extractor-tls" not found Sep 11 19:30:38.009356 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:38.009327 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qqtnn_447b6ece-d0e6-40cc-811c-faed4cdb4871/serve-healthcheck-canary/0.log" Sep 11 19:30:41.760783 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:41.760732 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:41.760783 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:41.760785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:41.761424 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:41.760935 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 19:30:41.761424 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:41.761017 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls podName:11495f40-d751-4bc7-8fc6-ed4d06014372 nodeName:}" failed. No retries permitted until 2026-09-11 19:30:49.760999174 +0000 UTC m=+283.033506560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rd8p6" (UID: "11495f40-d751-4bc7-8fc6-ed4d06014372") : secret "insights-runtime-extractor-tls" not found Sep 11 19:30:41.761424 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:41.761270 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11495f40-d751-4bc7-8fc6-ed4d06014372-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:47.772698 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:47.772659 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-zkfwb" podUID="06f12941-963e-4098-aeac-eb39ce1ea795" Sep 11 19:30:47.773188 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:47.772692 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" podUID="cc70d804-93b3-44c6-8102-6f325002eb8f" Sep 11 19:30:48.126388 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:48.126307 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zkfwb" Sep 11 19:30:48.126575 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:48.126307 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:30:49.827946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:49.827903 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:49.830308 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:49.830287 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11495f40-d751-4bc7-8fc6-ed4d06014372-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rd8p6\" (UID: \"11495f40-d751-4bc7-8fc6-ed4d06014372\") " pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:49.958155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:49.958122 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rd8p6" Sep 11 19:30:50.232047 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:50.231963 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rd8p6"] Sep 11 19:30:50.235806 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:30:50.235782 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11495f40_d751_4bc7_8fc6_ed4d06014372.slice/crio-209444d0463ec1aed051d63405b5f1875db429a11b13a29dd692ece275b35289 WatchSource:0}: Error finding container 209444d0463ec1aed051d63405b5f1875db429a11b13a29dd692ece275b35289: Status 404 returned error can't find the container with id 209444d0463ec1aed051d63405b5f1875db429a11b13a29dd692ece275b35289 Sep 11 19:30:51.137547 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.137502 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rd8p6" event={"ID":"11495f40-d751-4bc7-8fc6-ed4d06014372","Type":"ContainerStarted","Data":"8c1cfe98b5e9a09df81ca90c5df47be5142d8b1f242c8baa11393433b9c0ad1d"} Sep 11 19:30:51.137926 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.137609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rd8p6" event={"ID":"11495f40-d751-4bc7-8fc6-ed4d06014372","Type":"ContainerStarted","Data":"4e27a5d4639d253d8466fcb76b0db8e3b0b5da307199204021db0716e5785515"} Sep 11 19:30:51.137926 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.137640 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rd8p6" event={"ID":"11495f40-d751-4bc7-8fc6-ed4d06014372","Type":"ContainerStarted","Data":"209444d0463ec1aed051d63405b5f1875db429a11b13a29dd692ece275b35289"} Sep 11 19:30:51.138736 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.138714 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:30:51.138807 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.138794 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:30:51.141221 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.141202 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/06f12941-963e-4098-aeac-eb39ce1ea795-metrics-tls\") pod \"dns-default-zkfwb\" (UID: \"06f12941-963e-4098-aeac-eb39ce1ea795\") " pod="openshift-dns/dns-default-zkfwb" Sep 11 19:30:51.141479 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.141431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"image-registry-76bbf4cc86-47pdj\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:30:51.430117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.430088 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-x76hz\"" Sep 11 19:30:51.430117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.430087 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-p67zb\"" Sep 11 19:30:51.437193 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.437169 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zkfwb" Sep 11 19:30:51.437297 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.437243 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:30:51.720117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.720039 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zkfwb"] Sep 11 19:30:51.722941 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:51.722880 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76bbf4cc86-47pdj"] Sep 11 19:30:51.724001 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:30:51.723962 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f12941_963e_4098_aeac_eb39ce1ea795.slice/crio-eea13f3d5eba418137f2cf50d7124917fc8e5c1b6cbca6c5f338b7d589845f50 WatchSource:0}: Error finding container eea13f3d5eba418137f2cf50d7124917fc8e5c1b6cbca6c5f338b7d589845f50: Status 404 returned error can't find the container with id eea13f3d5eba418137f2cf50d7124917fc8e5c1b6cbca6c5f338b7d589845f50 Sep 11 19:30:51.728696 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:30:51.728671 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc70d804_93b3_44c6_8102_6f325002eb8f.slice/crio-673aa8209657573ee2752c481e007bed38d5dd7c875d2207f22517fe7160eeca WatchSource:0}: Error finding container 673aa8209657573ee2752c481e007bed38d5dd7c875d2207f22517fe7160eeca: Status 404 returned error can't find the container with id 673aa8209657573ee2752c481e007bed38d5dd7c875d2207f22517fe7160eeca Sep 11 19:30:51.783405 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:51.783375 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" podUID="3dbcdeec-40a0-4813-943f-172a623b04c5" Sep 11 19:30:52.143247 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:52.143210 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zkfwb" event={"ID":"06f12941-963e-4098-aeac-eb39ce1ea795","Type":"ContainerStarted","Data":"eea13f3d5eba418137f2cf50d7124917fc8e5c1b6cbca6c5f338b7d589845f50"} Sep 11 19:30:52.145068 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:52.145038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" event={"ID":"cc70d804-93b3-44c6-8102-6f325002eb8f","Type":"ContainerStarted","Data":"7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd"} Sep 11 19:30:52.145204 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:52.145071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:30:52.145204 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:52.145085 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" event={"ID":"cc70d804-93b3-44c6-8102-6f325002eb8f","Type":"ContainerStarted","Data":"673aa8209657573ee2752c481e007bed38d5dd7c875d2207f22517fe7160eeca"} Sep 11 19:30:52.166120 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:52.166073 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" podStartSLOduration=285.16605669 podStartE2EDuration="4m45.16605669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:52.164731706 +0000 UTC m=+285.437238903" watchObservedRunningTime="2026-09-11 19:30:52.16605669 +0000 UTC m=+285.438563889" Sep 11 19:30:53.150643 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:53.150603 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rd8p6" event={"ID":"11495f40-d751-4bc7-8fc6-ed4d06014372","Type":"ContainerStarted","Data":"bcea5c86f323c4667d391ef51e5a4a356085862ebbd9b9637326a69692a41c9a"} Sep 11 19:30:53.151167 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:53.150874 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:30:53.170280 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:53.170236 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rd8p6" podStartSLOduration=16.866920014 podStartE2EDuration="19.170219463s" totalImagesPullingTime="2.303299449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 19:30:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:53.169208304 +0000 UTC m=+286.441715511" watchObservedRunningTime="2026-09-11 19:30:53.170219463 +0000 UTC m=+286.442726663" Sep 11 19:30:54.155421 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:54.155378 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zkfwb" event={"ID":"06f12941-963e-4098-aeac-eb39ce1ea795","Type":"ContainerStarted","Data":"435deb37474d4e74489aee4d3466185e048e3f511d25214561ed52978c585151"} Sep 11 19:30:54.155914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:54.155456 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zkfwb" event={"ID":"06f12941-963e-4098-aeac-eb39ce1ea795","Type":"ContainerStarted","Data":"c630c6f15d664ffd14fd9e50b9111cbe87623432e5f2b4c9e94b9057e1ac7920"} Sep 11 19:30:54.155914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:54.155494 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zkfwb" Sep 11 19:30:54.172066 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:54.172021 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zkfwb" podStartSLOduration=251.320408375 podStartE2EDuration="4m13.172006211s" totalImagesPullingTime="1.851597836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:54.170733355 +0000 UTC m=+287.443240554" watchObservedRunningTime="2026-09-11 19:30:54.172006211 +0000 UTC m=+287.444513412" Sep 11 19:30:55.571112 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:55.571066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:30:55.573602 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:55.573579 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/3dbcdeec-40a0-4813-943f-172a623b04c5-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-94njz\" (UID: \"3dbcdeec-40a0-4813-943f-172a623b04c5\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:30:55.748721 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:55.748691 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qb6rt\"" Sep 11 19:30:55.756725 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:55.756709 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" Sep 11 19:30:56.026134 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:56.026101 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz"] Sep 11 19:30:56.029240 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:30:56.029208 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbcdeec_40a0_4813_943f_172a623b04c5.slice/crio-a62270459be0038962c4aa39e71bdcbe5f7805ed35d139196207969bb4503009 WatchSource:0}: Error finding container a62270459be0038962c4aa39e71bdcbe5f7805ed35d139196207969bb4503009: Status 404 returned error can't find the container with id a62270459be0038962c4aa39e71bdcbe5f7805ed35d139196207969bb4503009 Sep 11 19:30:56.162657 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:56.162624 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" event={"ID":"3dbcdeec-40a0-4813-943f-172a623b04c5","Type":"ContainerStarted","Data":"a62270459be0038962c4aa39e71bdcbe5f7805ed35d139196207969bb4503009"} Sep 11 19:30:57.168126 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:57.168098 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" event={"ID":"3dbcdeec-40a0-4813-943f-172a623b04c5","Type":"ContainerStarted","Data":"52c7bf0787b7ade1b81e4822570d8e22ba99310ca2d27f21f5a7345bc07812c5"} Sep 11 19:30:57.186048 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:30:57.186008 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-94njz" podStartSLOduration=251.1148851 podStartE2EDuration="4m12.185992789s" totalImagesPullingTime="1.071107689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:30:57.184927248 +0000 UTC m=+290.457434449" watchObservedRunningTime="2026-09-11 19:30:57.185992789 +0000 UTC m=+290.458500029" Sep 11 19:30:59.218769 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:30:59.218730 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-crcsn" podUID="0b2ec797-aa4b-4aa4-8472-782b11c2e9e4" Sep 11 19:31:00.715563 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:00.715521 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" podUID="363bfb12-8f9d-4a94-b51f-572c4f2ac4c9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 11 19:31:04.161569 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:04.161541 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zkfwb" Sep 11 19:31:07.135147 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:07.135117 2559 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 19:31:10.715644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:10.715606 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" podUID="363bfb12-8f9d-4a94-b51f-572c4f2ac4c9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 11 19:31:11.217929 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.217900 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:31:11.442510 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.442464 2559 patch_prober.go:28] interesting pod/image-registry-76bbf4cc86-47pdj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 19:31:11.442671 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.442516 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" podUID="cc70d804-93b3-44c6-8102-6f325002eb8f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 19:31:11.917618 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.917585 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fb8wd"] Sep 11 19:31:11.921177 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.921156 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:11.926814 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.926783 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 19:31:11.926937 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.926860 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 19:31:11.926937 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.926926 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 19:31:11.927077 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.927053 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 19:31:11.927772 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.927755 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 19:31:11.927859 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.927770 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nfn9q\"" Sep 11 19:31:11.927859 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:11.927758 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 19:31:12.085844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.085811 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-root\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.085844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.085853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.086057 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.085892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-textfile\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.086057 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.085943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bce0b0f-6048-4aa3-88a6-233b466228a0-metrics-client-ca\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.086057 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.085974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-tls\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.086199 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.086055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.086199 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.086098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-wtmp\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.086199 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.086117 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-sys\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.086199 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.086132 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhkj4\" (UniqueName: \"kubernetes.io/projected/5bce0b0f-6048-4aa3-88a6-233b466228a0-kube-api-access-hhkj4\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-root\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-textfile\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187601 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bce0b0f-6048-4aa3-88a6-233b466228a0-metrics-client-ca\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187570 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-root\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187636 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-tls\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187709 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187744 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-wtmp\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187996 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-sys\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187996 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187785 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhkj4\" (UniqueName: \"kubernetes.io/projected/5bce0b0f-6048-4aa3-88a6-233b466228a0-kube-api-access-hhkj4\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187996 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187844 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-sys\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187996 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187904 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-wtmp\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.187996 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.187939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-textfile\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.188297 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.188275 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-accelerators-collector-config\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.188363 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.188332 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5bce0b0f-6048-4aa3-88a6-233b466228a0-metrics-client-ca\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.190108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.190088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-tls\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.190195 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.190161 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5bce0b0f-6048-4aa3-88a6-233b466228a0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.196726 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.196699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhkj4\" (UniqueName: \"kubernetes.io/projected/5bce0b0f-6048-4aa3-88a6-233b466228a0-kube-api-access-hhkj4\") pod \"node-exporter-fb8wd\" (UID: \"5bce0b0f-6048-4aa3-88a6-233b466228a0\") " pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.230464 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.230412 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fb8wd" Sep 11 19:31:12.241007 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:31:12.240986 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bce0b0f_6048_4aa3_88a6_233b466228a0.slice/crio-88b803abd99dff442a9a11866b3ccd0b5acce5b1dbbec5be18c160fcd52e7956 WatchSource:0}: Error finding container 88b803abd99dff442a9a11866b3ccd0b5acce5b1dbbec5be18c160fcd52e7956: Status 404 returned error can't find the container with id 88b803abd99dff442a9a11866b3ccd0b5acce5b1dbbec5be18c160fcd52e7956 Sep 11 19:31:12.242833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.242815 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:31:12.976429 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.976392 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:31:12.985964 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.985929 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:12.988772 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.988744 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 19:31:12.988772 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.988760 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 19:31:12.989089 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.988830 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 19:31:12.989161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.989137 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 19:31:12.989519 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.989332 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 19:31:12.989519 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.989341 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 19:31:12.989519 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.989346 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 19:31:12.989519 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.989364 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xj7l6\"" Sep 11 19:31:12.989795 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.989749 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 19:31:12.996287 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.996066 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 19:31:12.997771 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:12.997746 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:31:13.093144 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093165 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093192 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093212 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnd5g\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-kube-api-access-gnd5g\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-config-out\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093295 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093627 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093403 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093627 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093488 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093627 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093538 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093627 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093583 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093627 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093606 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093848 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.093848 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.093683 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-web-config\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.194624 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.194811 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.194811 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194672 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.194811 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194695 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnd5g\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-kube-api-access-gnd5g\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.194811 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194733 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-config-out\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.194811 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.194811 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194789 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.195108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194817 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.195108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194846 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.195108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.195108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.195108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194935 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.195108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.194960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-web-config\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.196015 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.195688 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.196335 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.196304 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.196605 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.196542 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.198170 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.198122 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.198770 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.198728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.198890 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.198867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-web-config\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.199620 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.199558 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.199620 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.199578 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.200013 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.199989 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.200396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.200362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.200515 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.200479 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.200765 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.200745 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-config-out\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.205193 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.205170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnd5g\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-kube-api-access-gnd5g\") pod \"alertmanager-main-0\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.222150 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.222123 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fb8wd" event={"ID":"5bce0b0f-6048-4aa3-88a6-233b466228a0","Type":"ContainerStarted","Data":"88b803abd99dff442a9a11866b3ccd0b5acce5b1dbbec5be18c160fcd52e7956"} Sep 11 19:31:13.298334 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.298305 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:31:13.592153 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:13.592056 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:31:13.597664 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:31:13.597614 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88da152_97d2_461a_beb6_d1600f16768a.slice/crio-6be4a947f0aa77e01be862d68614f5ca596ed311fc36d27d0582309b3c412f82 WatchSource:0}: Error finding container 6be4a947f0aa77e01be862d68614f5ca596ed311fc36d27d0582309b3c412f82: Status 404 returned error can't find the container with id 6be4a947f0aa77e01be862d68614f5ca596ed311fc36d27d0582309b3c412f82 Sep 11 19:31:14.161028 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:14.160998 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:31:14.227683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:14.227647 2559 generic.go:356] "Generic (PLEG): container finished" podID="5bce0b0f-6048-4aa3-88a6-233b466228a0" containerID="4340cc2f6616c0c5ea4e4ca804d97f0aab6f8f1ac3d0f86d68edf12a6e17dc94" exitCode=0 Sep 11 19:31:14.227858 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:14.227781 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fb8wd" event={"ID":"5bce0b0f-6048-4aa3-88a6-233b466228a0","Type":"ContainerDied","Data":"4340cc2f6616c0c5ea4e4ca804d97f0aab6f8f1ac3d0f86d68edf12a6e17dc94"} Sep 11 19:31:14.229281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:14.229255 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerStarted","Data":"6be4a947f0aa77e01be862d68614f5ca596ed311fc36d27d0582309b3c412f82"} Sep 11 19:31:15.235115 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:15.235083 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fb8wd" event={"ID":"5bce0b0f-6048-4aa3-88a6-233b466228a0","Type":"ContainerStarted","Data":"dedfacf720e0f3c7a89d1195a489e1c6f864780f65b95ee9318cd7bb6d020fb2"} Sep 11 19:31:15.235568 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:15.235140 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fb8wd" event={"ID":"5bce0b0f-6048-4aa3-88a6-233b466228a0","Type":"ContainerStarted","Data":"3d9751743ef310093f0e4b5323da63ba674151058fd63676a08e098f24acc6d6"} Sep 11 19:31:15.236766 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:15.236723 2559 generic.go:356] "Generic (PLEG): container finished" podID="b88da152-97d2-461a-beb6-d1600f16768a" containerID="f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423" exitCode=0 Sep 11 19:31:15.236877 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:15.236798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerDied","Data":"f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423"} Sep 11 19:31:15.262623 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:15.262576 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fb8wd" podStartSLOduration=3.153410318 podStartE2EDuration="4.262561932s" totalImagesPullingTime="1.036739992s" totalInitContainerRuntime="72.411622ms" isStatefulPod=true podCreationTimestamp="2026-09-11 19:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:15.261793961 +0000 UTC m=+308.534301161" watchObservedRunningTime="2026-09-11 19:31:15.262561932 +0000 UTC m=+308.535069183" Sep 11 19:31:16.659303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.659268 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h"] Sep 11 19:31:16.666893 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.666867 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:16.669414 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.669338 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 19:31:16.669854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.669827 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qc7z8\"" Sep 11 19:31:16.670168 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.670119 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h"] Sep 11 19:31:16.723303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.723263 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7xw9h\" (UID: \"0b306842-5e77-4670-b2df-41b10871451b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:16.723488 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.723406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pbdj\" (UniqueName: \"kubernetes.io/projected/0b306842-5e77-4670-b2df-41b10871451b-kube-api-access-7pbdj\") pod \"monitoring-plugin-5d85778b9d-7xw9h\" (UID: \"0b306842-5e77-4670-b2df-41b10871451b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:16.824659 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.824625 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pbdj\" (UniqueName: \"kubernetes.io/projected/0b306842-5e77-4670-b2df-41b10871451b-kube-api-access-7pbdj\") pod \"monitoring-plugin-5d85778b9d-7xw9h\" (UID: \"0b306842-5e77-4670-b2df-41b10871451b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:16.824788 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.824684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7xw9h\" (UID: \"0b306842-5e77-4670-b2df-41b10871451b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:16.824834 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:31:16.824811 2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 11 19:31:16.824888 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:31:16.824876 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert podName:0b306842-5e77-4670-b2df-41b10871451b nodeName:}" failed. No retries permitted until 2026-09-11 19:31:17.324854696 +0000 UTC m=+310.597361901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-7xw9h" (UID: "0b306842-5e77-4670-b2df-41b10871451b") : secret "monitoring-plugin-cert" not found Sep 11 19:31:16.827136 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.827114 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76bbf4cc86-47pdj"] Sep 11 19:31:16.834368 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:16.834344 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pbdj\" (UniqueName: \"kubernetes.io/projected/0b306842-5e77-4670-b2df-41b10871451b-kube-api-access-7pbdj\") pod \"monitoring-plugin-5d85778b9d-7xw9h\" (UID: \"0b306842-5e77-4670-b2df-41b10871451b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:17.249987 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:17.249956 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerStarted","Data":"2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37"} Sep 11 19:31:17.250155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:17.250038 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerStarted","Data":"948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd"} Sep 11 19:31:17.250155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:17.250078 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerStarted","Data":"041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e"} Sep 11 19:31:17.250155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:17.250114 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerStarted","Data":"a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd"} Sep 11 19:31:17.250155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:17.250147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerStarted","Data":"6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d"} Sep 11 19:31:17.328680 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:17.328623 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7xw9h\" (UID: \"0b306842-5e77-4670-b2df-41b10871451b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:17.328849 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:31:17.328776 2559 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 11 19:31:17.328849 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:31:17.328839 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert podName:0b306842-5e77-4670-b2df-41b10871451b nodeName:}" failed. No retries permitted until 2026-09-11 19:31:18.328821638 +0000 UTC m=+311.601328816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-7xw9h" (UID: "0b306842-5e77-4670-b2df-41b10871451b") : secret "monitoring-plugin-cert" not found Sep 11 19:31:18.256629 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:18.256594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerStarted","Data":"ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd"} Sep 11 19:31:18.290344 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:18.290293 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.883269213 podStartE2EDuration="6.290272523s" totalImagesPullingTime="3.389282006s" totalInitContainerRuntime="17.721304ms" isStatefulPod=false podCreationTimestamp="2026-09-11 19:31:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:18.288788761 +0000 UTC m=+311.561295963" watchObservedRunningTime="2026-09-11 19:31:18.290272523 +0000 UTC m=+311.562779724" Sep 11 19:31:18.339760 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:18.339590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7xw9h\" (UID: \"0b306842-5e77-4670-b2df-41b10871451b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:18.342227 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:18.342199 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0b306842-5e77-4670-b2df-41b10871451b-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-7xw9h\" (UID: \"0b306842-5e77-4670-b2df-41b10871451b\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:18.478302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:18.478250 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:18.747928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:18.747900 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h"] Sep 11 19:31:18.750781 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:31:18.750752 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b306842_5e77_4670_b2df_41b10871451b.slice/crio-c81cf50404eaf03fbf512aae2eb1841cd426a730d48a02aba4d54f20bf6320df WatchSource:0}: Error finding container c81cf50404eaf03fbf512aae2eb1841cd426a730d48a02aba4d54f20bf6320df: Status 404 returned error can't find the container with id c81cf50404eaf03fbf512aae2eb1841cd426a730d48a02aba4d54f20bf6320df Sep 11 19:31:19.260859 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:19.260817 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" event={"ID":"0b306842-5e77-4670-b2df-41b10871451b","Type":"ContainerStarted","Data":"c81cf50404eaf03fbf512aae2eb1841cd426a730d48a02aba4d54f20bf6320df"} Sep 11 19:31:20.265656 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:20.265613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" event={"ID":"0b306842-5e77-4670-b2df-41b10871451b","Type":"ContainerStarted","Data":"d1c147f6dfcf0fdfb818052aa72d5e9006257382a2e7359c112c0ab1a8a87045"} Sep 11 19:31:20.266078 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:20.265843 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:20.270624 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:20.270595 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" Sep 11 19:31:20.282371 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:20.282328 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-7xw9h" podStartSLOduration=2.940783998 podStartE2EDuration="4.282316214s" totalImagesPullingTime="1.341532216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:31:20.281625447 +0000 UTC m=+313.554132647" watchObservedRunningTime="2026-09-11 19:31:20.282316214 +0000 UTC m=+313.554823461" Sep 11 19:31:20.715044 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:20.715007 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" podUID="363bfb12-8f9d-4a94-b51f-572c4f2ac4c9" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Sep 11 19:31:20.715221 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:20.715075 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" Sep 11 19:31:20.715585 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:20.715550 2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"4b41b58c4a86704e3f87f418eaa9d942edc40ad8c2fcd9fac1fecc6ccf49ba53"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" containerMessage="Container service-proxy failed liveness probe, will be restarted" Sep 11 19:31:20.715660 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:20.715604 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" podUID="363bfb12-8f9d-4a94-b51f-572c4f2ac4c9" containerName="service-proxy" containerID="cri-o://4b41b58c4a86704e3f87f418eaa9d942edc40ad8c2fcd9fac1fecc6ccf49ba53" gracePeriod=30 Sep 11 19:31:21.271770 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:21.271742 2559 generic.go:356] "Generic (PLEG): container finished" podID="363bfb12-8f9d-4a94-b51f-572c4f2ac4c9" containerID="4b41b58c4a86704e3f87f418eaa9d942edc40ad8c2fcd9fac1fecc6ccf49ba53" exitCode=2 Sep 11 19:31:21.272127 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:21.271828 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" event={"ID":"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9","Type":"ContainerDied","Data":"4b41b58c4a86704e3f87f418eaa9d942edc40ad8c2fcd9fac1fecc6ccf49ba53"} Sep 11 19:31:21.272127 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:21.271880 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-c4858f78f-q6mzp" event={"ID":"363bfb12-8f9d-4a94-b51f-572c4f2ac4c9","Type":"ContainerStarted","Data":"e919a8db548089246781bb45c1d7409305889af9e56eba4810d5bdbc5592cc25"} Sep 11 19:31:41.852502 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:41.852415 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" podUID="cc70d804-93b3-44c6-8102-6f325002eb8f" containerName="registry" containerID="cri-o://7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd" gracePeriod=30 Sep 11 19:31:42.098589 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.098566 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:31:42.237495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.237426 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-image-registry-private-configuration\") pod \"cc70d804-93b3-44c6-8102-6f325002eb8f\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " Sep 11 19:31:42.237495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.237481 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-bound-sa-token\") pod \"cc70d804-93b3-44c6-8102-6f325002eb8f\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " Sep 11 19:31:42.237692 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.237521 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k22cg\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-kube-api-access-k22cg\") pod \"cc70d804-93b3-44c6-8102-6f325002eb8f\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " Sep 11 19:31:42.237692 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.237552 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-installation-pull-secrets\") pod \"cc70d804-93b3-44c6-8102-6f325002eb8f\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " Sep 11 19:31:42.237692 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.237575 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") pod \"cc70d804-93b3-44c6-8102-6f325002eb8f\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " Sep 11 19:31:42.237692 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.237610 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-trusted-ca\") pod \"cc70d804-93b3-44c6-8102-6f325002eb8f\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " Sep 11 19:31:42.237885 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.237769 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-certificates\") pod \"cc70d804-93b3-44c6-8102-6f325002eb8f\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " Sep 11 19:31:42.237885 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.237822 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc70d804-93b3-44c6-8102-6f325002eb8f-ca-trust-extracted\") pod \"cc70d804-93b3-44c6-8102-6f325002eb8f\" (UID: \"cc70d804-93b3-44c6-8102-6f325002eb8f\") " Sep 11 19:31:42.238105 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.238052 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cc70d804-93b3-44c6-8102-6f325002eb8f" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:31:42.238504 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.238395 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cc70d804-93b3-44c6-8102-6f325002eb8f" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:31:42.240284 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.240252 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-kube-api-access-k22cg" (OuterVolumeSpecName: "kube-api-access-k22cg") pod "cc70d804-93b3-44c6-8102-6f325002eb8f" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f"). InnerVolumeSpecName "kube-api-access-k22cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:31:42.240472 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.240371 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cc70d804-93b3-44c6-8102-6f325002eb8f" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:31:42.240531 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.240481 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cc70d804-93b3-44c6-8102-6f325002eb8f" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:31:42.240531 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.240503 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cc70d804-93b3-44c6-8102-6f325002eb8f" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:31:42.240611 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.240583 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cc70d804-93b3-44c6-8102-6f325002eb8f" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:31:42.248756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.248732 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc70d804-93b3-44c6-8102-6f325002eb8f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cc70d804-93b3-44c6-8102-6f325002eb8f" (UID: "cc70d804-93b3-44c6-8102-6f325002eb8f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:31:42.338726 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.338686 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-installation-pull-secrets\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:31:42.338726 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.338722 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-tls\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:31:42.338914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.338737 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-trusted-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:31:42.338914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.338751 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cc70d804-93b3-44c6-8102-6f325002eb8f-registry-certificates\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:31:42.338914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.338765 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cc70d804-93b3-44c6-8102-6f325002eb8f-ca-trust-extracted\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:31:42.338914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.338775 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cc70d804-93b3-44c6-8102-6f325002eb8f-image-registry-private-configuration\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:31:42.338914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.338782 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-bound-sa-token\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:31:42.338914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.338790 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k22cg\" (UniqueName: \"kubernetes.io/projected/cc70d804-93b3-44c6-8102-6f325002eb8f-kube-api-access-k22cg\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:31:42.342609 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.342582 2559 generic.go:356] "Generic (PLEG): container finished" podID="cc70d804-93b3-44c6-8102-6f325002eb8f" containerID="7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd" exitCode=0 Sep 11 19:31:42.342666 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.342627 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" Sep 11 19:31:42.342666 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.342637 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" event={"ID":"cc70d804-93b3-44c6-8102-6f325002eb8f","Type":"ContainerDied","Data":"7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd"} Sep 11 19:31:42.342746 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.342685 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76bbf4cc86-47pdj" event={"ID":"cc70d804-93b3-44c6-8102-6f325002eb8f","Type":"ContainerDied","Data":"673aa8209657573ee2752c481e007bed38d5dd7c875d2207f22517fe7160eeca"} Sep 11 19:31:42.342746 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.342713 2559 scope.go:117] "RemoveContainer" containerID="7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd" Sep 11 19:31:42.351079 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.351062 2559 scope.go:117] "RemoveContainer" containerID="7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd" Sep 11 19:31:42.351339 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:31:42.351317 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd\": container with ID starting with 7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd not found: ID does not exist" containerID="7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd" Sep 11 19:31:42.351391 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.351348 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd"} err="failed to get container status \"7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd\": rpc error: code = NotFound desc = could not find container \"7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd\": container with ID starting with 7c1882c1a0ec25022bbf6b742e418e5fd293ec690058fcc998f785b5fc9fe8fd not found: ID does not exist" Sep 11 19:31:42.362404 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.362383 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76bbf4cc86-47pdj"] Sep 11 19:31:42.366383 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:42.366364 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76bbf4cc86-47pdj"] Sep 11 19:31:43.221688 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:31:43.221658 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc70d804-93b3-44c6-8102-6f325002eb8f" path="/var/lib/kubelet/pods/cc70d804-93b3-44c6-8102-6f325002eb8f/volumes" Sep 11 19:32:21.128346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:21.128307 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:32:21.130646 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:21.130626 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b2ec797-aa4b-4aa4-8472-782b11c2e9e4-metrics-certs\") pod \"network-metrics-daemon-crcsn\" (UID: \"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4\") " pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:32:21.422358 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:21.422279 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k4fsc\"" Sep 11 19:32:21.429752 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:21.429734 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-crcsn" Sep 11 19:32:21.701829 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:21.701754 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-crcsn"] Sep 11 19:32:21.704860 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:32:21.704828 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b2ec797_aa4b_4aa4_8472_782b11c2e9e4.slice/crio-3f367891a389fd8525ac7a16475f74dac5d00830545f8d53d42001cd4ae904d5 WatchSource:0}: Error finding container 3f367891a389fd8525ac7a16475f74dac5d00830545f8d53d42001cd4ae904d5: Status 404 returned error can't find the container with id 3f367891a389fd8525ac7a16475f74dac5d00830545f8d53d42001cd4ae904d5 Sep 11 19:32:22.485685 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:22.485635 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crcsn" event={"ID":"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4","Type":"ContainerStarted","Data":"3f367891a389fd8525ac7a16475f74dac5d00830545f8d53d42001cd4ae904d5"} Sep 11 19:32:23.490557 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:23.490518 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crcsn" event={"ID":"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4","Type":"ContainerStarted","Data":"3dcf2f9b2c31ab89f0a364e2b024be130852bddbc6685926169f571cd136fd6e"} Sep 11 19:32:23.490923 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:23.490577 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-crcsn" event={"ID":"0b2ec797-aa4b-4aa4-8472-782b11c2e9e4","Type":"ContainerStarted","Data":"4e8862af4ae69ccebafa1b93e5d01778963a99ebb616386dba69f581ad104127"} Sep 11 19:32:23.506419 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:23.506375 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-crcsn" podStartSLOduration=375.439509735 podStartE2EDuration="6m16.50635506s" totalImagesPullingTime="1.066845325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:32:23.505875772 +0000 UTC m=+376.778382973" watchObservedRunningTime="2026-09-11 19:32:23.50635506 +0000 UTC m=+376.778862445" Sep 11 19:32:32.168368 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.168327 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:32:32.168867 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.168681 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="alertmanager" containerID="cri-o://6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d" gracePeriod=120 Sep 11 19:32:32.168867 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.168754 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy-metric" containerID="cri-o://2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37" gracePeriod=120 Sep 11 19:32:32.168867 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.168789 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="config-reloader" containerID="cri-o://a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd" gracePeriod=120 Sep 11 19:32:32.168867 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.168821 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="prom-label-proxy" containerID="cri-o://ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd" gracePeriod=120 Sep 11 19:32:32.169111 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.168759 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy-web" containerID="cri-o://041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e" gracePeriod=120 Sep 11 19:32:32.169691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.168997 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy" containerID="cri-o://948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd" gracePeriod=120 Sep 11 19:32:32.524566 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524536 2559 generic.go:356] "Generic (PLEG): container finished" podID="b88da152-97d2-461a-beb6-d1600f16768a" containerID="ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd" exitCode=0 Sep 11 19:32:32.524566 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524559 2559 generic.go:356] "Generic (PLEG): container finished" podID="b88da152-97d2-461a-beb6-d1600f16768a" containerID="2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37" exitCode=0 Sep 11 19:32:32.524566 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524566 2559 generic.go:356] "Generic (PLEG): container finished" podID="b88da152-97d2-461a-beb6-d1600f16768a" containerID="948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd" exitCode=0 Sep 11 19:32:32.524566 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524571 2559 generic.go:356] "Generic (PLEG): container finished" podID="b88da152-97d2-461a-beb6-d1600f16768a" containerID="a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd" exitCode=0 Sep 11 19:32:32.524566 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524576 2559 generic.go:356] "Generic (PLEG): container finished" podID="b88da152-97d2-461a-beb6-d1600f16768a" containerID="6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d" exitCode=0 Sep 11 19:32:32.524913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524607 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerDied","Data":"ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd"} Sep 11 19:32:32.524913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524686 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerDied","Data":"2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37"} Sep 11 19:32:32.524913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524724 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerDied","Data":"948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd"} Sep 11 19:32:32.524913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524759 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerDied","Data":"a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd"} Sep 11 19:32:32.524913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:32.524799 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerDied","Data":"6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d"} Sep 11 19:32:33.410111 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.410083 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.513713 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.513684 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.513888 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.513721 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-main-db\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.513888 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.513739 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.513888 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.513761 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-config-volume\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.514041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.513891 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-metrics-client-ca\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.514041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.513932 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnd5g\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-kube-api-access-gnd5g\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.514041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.513969 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-web\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.514041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.513997 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-cluster-tls-config\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.514041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.514037 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-config-out\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.514281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.514057 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:32:33.514281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.514076 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-main-tls\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.514281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.514102 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-trusted-ca-bundle\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.514281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.514147 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-tls-assets\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.515683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.514972 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:32:33.515683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.515028 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-web-config\") pod \"b88da152-97d2-461a-beb6-d1600f16768a\" (UID: \"b88da152-97d2-461a-beb6-d1600f16768a\") " Sep 11 19:32:33.515683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.515620 2559 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-metrics-client-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.515683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.515643 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-main-db\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.515683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.515615 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:32:33.517136 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.516837 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-config-volume" (OuterVolumeSpecName: "config-volume") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:32:33.517252 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.517153 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:32:33.517318 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.517260 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:32:33.517318 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.517284 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:32:33.517736 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.517696 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:32:33.518008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.517987 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:32:33.518386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.518361 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-kube-api-access-gnd5g" (OuterVolumeSpecName: "kube-api-access-gnd5g") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "kube-api-access-gnd5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:32:33.518669 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.518649 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-config-out" (OuterVolumeSpecName: "config-out") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:32:33.520756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.520730 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:32:33.527551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.527517 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-web-config" (OuterVolumeSpecName: "web-config") pod "b88da152-97d2-461a-beb6-d1600f16768a" (UID: "b88da152-97d2-461a-beb6-d1600f16768a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:32:33.532426 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.532406 2559 generic.go:356] "Generic (PLEG): container finished" podID="b88da152-97d2-461a-beb6-d1600f16768a" containerID="041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e" exitCode=0 Sep 11 19:32:33.532535 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.532479 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerDied","Data":"041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e"} Sep 11 19:32:33.532535 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.532510 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.532650 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.532572 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b88da152-97d2-461a-beb6-d1600f16768a","Type":"ContainerDied","Data":"6be4a947f0aa77e01be862d68614f5ca596ed311fc36d27d0582309b3c412f82"} Sep 11 19:32:33.532650 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.532626 2559 scope.go:117] "RemoveContainer" containerID="ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd" Sep 11 19:32:33.540262 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.540246 2559 scope.go:117] "RemoveContainer" containerID="2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37" Sep 11 19:32:33.547573 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.547555 2559 scope.go:117] "RemoveContainer" containerID="948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd" Sep 11 19:32:33.554482 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.554460 2559 scope.go:117] "RemoveContainer" containerID="041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e" Sep 11 19:32:33.554827 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.554807 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:32:33.561164 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.561136 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:32:33.561792 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.561777 2559 scope.go:117] "RemoveContainer" containerID="a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd" Sep 11 19:32:33.568249 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.568232 2559 scope.go:117] "RemoveContainer" containerID="6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d" Sep 11 19:32:33.574769 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.574739 2559 scope.go:117] "RemoveContainer" containerID="f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423" Sep 11 19:32:33.581402 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.581388 2559 scope.go:117] "RemoveContainer" containerID="ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd" Sep 11 19:32:33.581679 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:32:33.581663 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd\": container with ID starting with ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd not found: ID does not exist" containerID="ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd" Sep 11 19:32:33.581725 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.581687 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd"} err="failed to get container status \"ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd\": rpc error: code = NotFound desc = could not find container \"ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd\": container with ID starting with ed7a42275de4d533f93e018e5903d0c5c4512d165589ce50831df675cb2918dd not found: ID does not exist" Sep 11 19:32:33.581725 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.581704 2559 scope.go:117] "RemoveContainer" containerID="2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37" Sep 11 19:32:33.581965 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:32:33.581945 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37\": container with ID starting with 2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37 not found: ID does not exist" containerID="2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37" Sep 11 19:32:33.582031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.581974 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37"} err="failed to get container status \"2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37\": rpc error: code = NotFound desc = could not find container \"2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37\": container with ID starting with 2bdb05e7aa604d3c3cc925282dc15c9d554263d1591c80da67a3c76b3e82cd37 not found: ID does not exist" Sep 11 19:32:33.582031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.581997 2559 scope.go:117] "RemoveContainer" containerID="948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd" Sep 11 19:32:33.582249 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:32:33.582234 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd\": container with ID starting with 948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd not found: ID does not exist" containerID="948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd" Sep 11 19:32:33.582310 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.582255 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd"} err="failed to get container status \"948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd\": rpc error: code = NotFound desc = could not find container \"948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd\": container with ID starting with 948bde80f1dd27b68e052ae23d2be1d74d51f9aa7d9301f6de9e4f5d192798fd not found: ID does not exist" Sep 11 19:32:33.582310 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.582275 2559 scope.go:117] "RemoveContainer" containerID="041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e" Sep 11 19:32:33.582543 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:32:33.582525 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e\": container with ID starting with 041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e not found: ID does not exist" containerID="041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e" Sep 11 19:32:33.582543 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.582545 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e"} err="failed to get container status \"041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e\": rpc error: code = NotFound desc = could not find container \"041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e\": container with ID starting with 041e330ff79036afc7e841a6bff1b5c8a1f135190f353c5b6d1e40e66682740e not found: ID does not exist" Sep 11 19:32:33.582687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.582558 2559 scope.go:117] "RemoveContainer" containerID="a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd" Sep 11 19:32:33.582796 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:32:33.582780 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd\": container with ID starting with a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd not found: ID does not exist" containerID="a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd" Sep 11 19:32:33.582838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.582799 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd"} err="failed to get container status \"a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd\": rpc error: code = NotFound desc = could not find container \"a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd\": container with ID starting with a4eabf90f442200c5bad88874f19e14d5b1ed6620874616db2fd82132a2944fd not found: ID does not exist" Sep 11 19:32:33.582838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.582819 2559 scope.go:117] "RemoveContainer" containerID="6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d" Sep 11 19:32:33.583032 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:32:33.583015 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d\": container with ID starting with 6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d not found: ID does not exist" containerID="6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d" Sep 11 19:32:33.583071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.583035 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d"} err="failed to get container status \"6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d\": rpc error: code = NotFound desc = could not find container \"6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d\": container with ID starting with 6bbbbaa44b245e6a882bc4b59f89605d63b0ea39872ecfc65d63d7c0ecbd866d not found: ID does not exist" Sep 11 19:32:33.583071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.583048 2559 scope.go:117] "RemoveContainer" containerID="f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423" Sep 11 19:32:33.583255 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:32:33.583237 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423\": container with ID starting with f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423 not found: ID does not exist" containerID="f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423" Sep 11 19:32:33.583293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.583261 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423"} err="failed to get container status \"f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423\": rpc error: code = NotFound desc = could not find container \"f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423\": container with ID starting with f19d8a8312a568df6a72470bc121fac83f56d51f762c7d5218dd64e798876423 not found: ID does not exist" Sep 11 19:32:33.589849 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.589830 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:32:33.590062 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590051 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="init-config-reloader" Sep 11 19:32:33.590098 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590063 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="init-config-reloader" Sep 11 19:32:33.590098 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590071 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy" Sep 11 19:32:33.590098 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590077 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy" Sep 11 19:32:33.590098 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590085 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="prom-label-proxy" Sep 11 19:32:33.590098 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590090 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="prom-label-proxy" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590101 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="alertmanager" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590106 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="alertmanager" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590112 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc70d804-93b3-44c6-8102-6f325002eb8f" containerName="registry" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590117 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc70d804-93b3-44c6-8102-6f325002eb8f" containerName="registry" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590123 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy-metric" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590128 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy-metric" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590136 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="config-reloader" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590141 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="config-reloader" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590146 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy-web" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590151 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy-web" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590188 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="config-reloader" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590195 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy-web" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590200 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="prom-label-proxy" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590205 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590211 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="alertmanager" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590216 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc70d804-93b3-44c6-8102-6f325002eb8f" containerName="registry" Sep 11 19:32:33.590242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.590220 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b88da152-97d2-461a-beb6-d1600f16768a" containerName="kube-rbac-proxy-metric" Sep 11 19:32:33.593123 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.593108 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.598248 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.598229 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 19:32:33.598248 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.598243 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 19:32:33.598400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.598234 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 19:32:33.598400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.598357 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 19:32:33.598549 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.598532 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 19:32:33.598831 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.598810 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 19:32:33.598945 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.598817 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 19:32:33.598945 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.598903 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-xj7l6\"" Sep 11 19:32:33.599160 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.599003 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 19:32:33.604077 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.604060 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 19:32:33.613213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.613188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:32:33.616276 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616253 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616362 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616280 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-web-config\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616362 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da9c672e-9851-4719-8809-d54d98d1dd5b-config-out\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616362 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616312 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616382 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da9c672e-9851-4719-8809-d54d98d1dd5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616460 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616493 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da9c672e-9851-4719-8809-d54d98d1dd5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616517 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp7zv\" (UniqueName: \"kubernetes.io/projected/da9c672e-9851-4719-8809-d54d98d1dd5b-kube-api-access-hp7zv\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616572 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9c672e-9851-4719-8809-d54d98d1dd5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616590 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c672e-9851-4719-8809-d54d98d1dd5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616644 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616695 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616705 2559 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-cluster-tls-config\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616716 2559 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b88da152-97d2-461a-beb6-d1600f16768a-config-out\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616725 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-main-tls\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616733 2559 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b88da152-97d2-461a-beb6-d1600f16768a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616741 2559 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-tls-assets\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616748 2559 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-web-config\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616757 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616765 2559 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616773 2559 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b88da152-97d2-461a-beb6-d1600f16768a-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.616822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.616782 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnd5g\" (UniqueName: \"kubernetes.io/projected/b88da152-97d2-461a-beb6-d1600f16768a-kube-api-access-gnd5g\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:32:33.717300 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da9c672e-9851-4719-8809-d54d98d1dd5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717300 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717303 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp7zv\" (UniqueName: \"kubernetes.io/projected/da9c672e-9851-4719-8809-d54d98d1dd5b-kube-api-access-hp7zv\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717326 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9c672e-9851-4719-8809-d54d98d1dd5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c672e-9851-4719-8809-d54d98d1dd5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717696 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717696 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717696 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717696 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717659 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-web-config\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717696 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da9c672e-9851-4719-8809-d54d98d1dd5b-config-out\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717718 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717755 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da9c672e-9851-4719-8809-d54d98d1dd5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.717955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.717813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.718337 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.718283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da9c672e-9851-4719-8809-d54d98d1dd5b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.718636 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.718607 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da9c672e-9851-4719-8809-d54d98d1dd5b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.720659 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.720631 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da9c672e-9851-4719-8809-d54d98d1dd5b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.720880 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.720792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.720982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.720900 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.721056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.720988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-config-volume\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.721118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.721050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.721206 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.721180 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.721322 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.721297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da9c672e-9851-4719-8809-d54d98d1dd5b-config-out\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.721372 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.721337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da9c672e-9851-4719-8809-d54d98d1dd5b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.721513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.721497 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.722706 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.722687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da9c672e-9851-4719-8809-d54d98d1dd5b-web-config\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.725203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.725184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp7zv\" (UniqueName: \"kubernetes.io/projected/da9c672e-9851-4719-8809-d54d98d1dd5b-kube-api-access-hp7zv\") pod \"alertmanager-main-0\" (UID: \"da9c672e-9851-4719-8809-d54d98d1dd5b\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:33.901804 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:33.901699 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 19:32:34.181261 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:34.181185 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 19:32:34.184569 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:32:34.184544 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9c672e_9851_4719_8809_d54d98d1dd5b.slice/crio-ed393b17b64567760fa8f034821a89ab78657409c5ce8b4f43412a9140561495 WatchSource:0}: Error finding container ed393b17b64567760fa8f034821a89ab78657409c5ce8b4f43412a9140561495: Status 404 returned error can't find the container with id ed393b17b64567760fa8f034821a89ab78657409c5ce8b4f43412a9140561495 Sep 11 19:32:34.538653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:34.538618 2559 generic.go:356] "Generic (PLEG): container finished" podID="da9c672e-9851-4719-8809-d54d98d1dd5b" containerID="2e6b9868b3bf156bf06ce30ec3612e99a265a58ad3b329bbce8a10013d00cfb6" exitCode=0 Sep 11 19:32:34.539132 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:34.538674 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da9c672e-9851-4719-8809-d54d98d1dd5b","Type":"ContainerDied","Data":"2e6b9868b3bf156bf06ce30ec3612e99a265a58ad3b329bbce8a10013d00cfb6"} Sep 11 19:32:34.539132 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:34.538748 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da9c672e-9851-4719-8809-d54d98d1dd5b","Type":"ContainerStarted","Data":"ed393b17b64567760fa8f034821a89ab78657409c5ce8b4f43412a9140561495"} Sep 11 19:32:35.222318 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:35.222247 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b88da152-97d2-461a-beb6-d1600f16768a" path="/var/lib/kubelet/pods/b88da152-97d2-461a-beb6-d1600f16768a/volumes" Sep 11 19:32:35.546703 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:35.546669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da9c672e-9851-4719-8809-d54d98d1dd5b","Type":"ContainerStarted","Data":"e4c4f09b352eaa63179c23e665daffa0287b2d2b80f8c8f665f44aa915cdab06"} Sep 11 19:32:35.547118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:35.546734 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da9c672e-9851-4719-8809-d54d98d1dd5b","Type":"ContainerStarted","Data":"3dcaca8ec453ad665f50daf31a2a029885f8c44d0608ec876b1fca62a11a9631"} Sep 11 19:32:35.547118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:35.546761 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da9c672e-9851-4719-8809-d54d98d1dd5b","Type":"ContainerStarted","Data":"a08b02b04ea086e5382565bfb1cc5ee7fda10e567b2fd7e00910f88763193afe"} Sep 11 19:32:35.547118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:35.546784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da9c672e-9851-4719-8809-d54d98d1dd5b","Type":"ContainerStarted","Data":"776bd938027e2aa17d531a017942cf614c7d9b74281102d1c530bced6034a2ff"} Sep 11 19:32:35.547118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:35.546822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da9c672e-9851-4719-8809-d54d98d1dd5b","Type":"ContainerStarted","Data":"122edd4bbaaaa1a6a15bf404c1492a4fafb08abd2610d01fde25821f9e620f87"} Sep 11 19:32:35.547118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:35.546849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da9c672e-9851-4719-8809-d54d98d1dd5b","Type":"ContainerStarted","Data":"ed25eaa7e2326bcb2c8fcc47194049ba78c7903725327544ed1ec4b0910bc34e"} Sep 11 19:32:35.577744 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:35.577673 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.560740057 podStartE2EDuration="2.577652894s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.912837ms" isStatefulPod=false podCreationTimestamp="2026-09-11 19:32:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:32:35.576069998 +0000 UTC m=+388.848577199" watchObservedRunningTime="2026-09-11 19:32:35.577652894 +0000 UTC m=+388.850160099" Sep 11 19:32:36.180597 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.180563 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-55d94b4764-2vb6k"] Sep 11 19:32:36.184107 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.184084 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.187056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.187033 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 19:32:36.187164 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.187059 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 19:32:36.187286 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.187273 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 19:32:36.187417 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.187392 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-h674d\"" Sep 11 19:32:36.187563 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.187464 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 19:32:36.187563 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.187483 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 19:32:36.192732 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.192711 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 19:32:36.196122 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.196100 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55d94b4764-2vb6k"] Sep 11 19:32:36.236474 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.236429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-metrics-client-ca\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.236603 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.236500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.236603 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.236525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.236603 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.236543 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-secret-telemeter-client\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.236603 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.236568 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-federate-client-tls\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.236730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.236662 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dvm5\" (UniqueName: \"kubernetes.io/projected/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-kube-api-access-6dvm5\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.236730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.236679 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-serving-certs-ca-bundle\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.236730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.236708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-telemeter-client-tls\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.337550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.337518 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dvm5\" (UniqueName: \"kubernetes.io/projected/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-kube-api-access-6dvm5\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.337550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.337551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-serving-certs-ca-bundle\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.337704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.337578 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-telemeter-client-tls\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.337704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.337600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-metrics-client-ca\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.337704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.337633 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.337704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.337665 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.337704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.337689 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-secret-telemeter-client\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.337931 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.337734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-federate-client-tls\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.338540 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.338424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-metrics-client-ca\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.338540 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.338533 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.338749 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.338574 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-serving-certs-ca-bundle\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.340292 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.340267 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-federate-client-tls\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.340396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.340308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.340739 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.340718 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-secret-telemeter-client\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.340870 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.340850 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-telemeter-client-tls\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.346073 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.346052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dvm5\" (UniqueName: \"kubernetes.io/projected/8d2faaf3-fabb-4c49-a88f-112010fa4e0f-kube-api-access-6dvm5\") pod \"telemeter-client-55d94b4764-2vb6k\" (UID: \"8d2faaf3-fabb-4c49-a88f-112010fa4e0f\") " pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.494199 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.494169 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" Sep 11 19:32:36.774248 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:36.774130 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-55d94b4764-2vb6k"] Sep 11 19:32:36.776213 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:32:36.776188 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2faaf3_fabb_4c49_a88f_112010fa4e0f.slice/crio-cac81dd0e7d397a46ec4555b6ab3c5a880e4a7cbaa56dc6a5a74de2123a3dd5d WatchSource:0}: Error finding container cac81dd0e7d397a46ec4555b6ab3c5a880e4a7cbaa56dc6a5a74de2123a3dd5d: Status 404 returned error can't find the container with id cac81dd0e7d397a46ec4555b6ab3c5a880e4a7cbaa56dc6a5a74de2123a3dd5d Sep 11 19:32:37.559378 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:37.559341 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" event={"ID":"8d2faaf3-fabb-4c49-a88f-112010fa4e0f","Type":"ContainerStarted","Data":"cac81dd0e7d397a46ec4555b6ab3c5a880e4a7cbaa56dc6a5a74de2123a3dd5d"} Sep 11 19:32:39.569761 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:39.569724 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" event={"ID":"8d2faaf3-fabb-4c49-a88f-112010fa4e0f","Type":"ContainerStarted","Data":"7c3bbf9aeb7380177f8824d7a7780491545236862674b199c45ff8a2c143ad26"} Sep 11 19:32:39.570141 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:39.569791 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" event={"ID":"8d2faaf3-fabb-4c49-a88f-112010fa4e0f","Type":"ContainerStarted","Data":"847e9d8560497a64d386ad19465365b0388e22d070008ff3078701d5f93697d3"} Sep 11 19:32:39.570141 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:39.569830 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" event={"ID":"8d2faaf3-fabb-4c49-a88f-112010fa4e0f","Type":"ContainerStarted","Data":"3ecee5be0ec112a4a305a8d118590c615faa6eecbd4009d95e04b6137c6237bd"} Sep 11 19:32:39.592430 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:32:39.592380 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-55d94b4764-2vb6k" podStartSLOduration=1.639727054 podStartE2EDuration="3.592365231s" totalImagesPullingTime="1.952638177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:32:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:32:39.590069577 +0000 UTC m=+392.862576777" watchObservedRunningTime="2026-09-11 19:32:39.592365231 +0000 UTC m=+392.864872431" Sep 11 19:34:38.051039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.051007 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz"] Sep 11 19:34:38.054373 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.054349 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.057338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.057317 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p9dlz\"" Sep 11 19:34:38.057538 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.057516 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 19:34:38.058322 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.058303 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 19:34:38.062050 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.062029 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz"] Sep 11 19:34:38.125100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.125067 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.125270 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.125124 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.125270 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.125144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfvm2\" (UniqueName: \"kubernetes.io/projected/9011b3c2-ae6c-4252-bcce-594ffca54aef-kube-api-access-zfvm2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.225963 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.225937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.225963 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.225965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfvm2\" (UniqueName: \"kubernetes.io/projected/9011b3c2-ae6c-4252-bcce-594ffca54aef-kube-api-access-zfvm2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.226176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.225995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.226340 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.226318 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.226340 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.226331 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.234308 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.234284 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfvm2\" (UniqueName: \"kubernetes.io/projected/9011b3c2-ae6c-4252-bcce-594ffca54aef-kube-api-access-zfvm2\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.364614 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.364532 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:38.637938 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.637914 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz"] Sep 11 19:34:38.640240 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:34:38.640213 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9011b3c2_ae6c_4252_bcce_594ffca54aef.slice/crio-443197eaa019489185770a805af0ea14c0a3a3e3ae77eda9599d697a5446fc37 WatchSource:0}: Error finding container 443197eaa019489185770a805af0ea14c0a3a3e3ae77eda9599d697a5446fc37: Status 404 returned error can't find the container with id 443197eaa019489185770a805af0ea14c0a3a3e3ae77eda9599d697a5446fc37 Sep 11 19:34:38.997517 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:38.997480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" event={"ID":"9011b3c2-ae6c-4252-bcce-594ffca54aef","Type":"ContainerStarted","Data":"443197eaa019489185770a805af0ea14c0a3a3e3ae77eda9599d697a5446fc37"} Sep 11 19:34:47.030016 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:47.029984 2559 generic.go:356] "Generic (PLEG): container finished" podID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerID="1434285a26c7756a7a786c1dc28e5f5125b742d80d3c96c67e9ba15c5a7611c3" exitCode=0 Sep 11 19:34:47.030421 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:47.030031 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" event={"ID":"9011b3c2-ae6c-4252-bcce-594ffca54aef","Type":"ContainerDied","Data":"1434285a26c7756a7a786c1dc28e5f5125b742d80d3c96c67e9ba15c5a7611c3"} Sep 11 19:34:50.043594 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:50.043555 2559 generic.go:356] "Generic (PLEG): container finished" podID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerID="9a2f27b70dab0c5dd96181ef7a4413ae64fc899b0beee6c2bc86a556eecb90a3" exitCode=0 Sep 11 19:34:50.043959 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:50.043605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" event={"ID":"9011b3c2-ae6c-4252-bcce-594ffca54aef","Type":"ContainerDied","Data":"9a2f27b70dab0c5dd96181ef7a4413ae64fc899b0beee6c2bc86a556eecb90a3"} Sep 11 19:34:58.077410 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:58.077383 2559 generic.go:356] "Generic (PLEG): container finished" podID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerID="484876b1a6f51e4593377fcd93ea782ebb05598ebd898313a31e901009ddd310" exitCode=0 Sep 11 19:34:58.077779 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:58.077464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" event={"ID":"9011b3c2-ae6c-4252-bcce-594ffca54aef","Type":"ContainerDied","Data":"484876b1a6f51e4593377fcd93ea782ebb05598ebd898313a31e901009ddd310"} Sep 11 19:34:59.205650 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.205628 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:34:59.305332 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.305294 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-util\") pod \"9011b3c2-ae6c-4252-bcce-594ffca54aef\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " Sep 11 19:34:59.305332 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.305337 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-bundle\") pod \"9011b3c2-ae6c-4252-bcce-594ffca54aef\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " Sep 11 19:34:59.305568 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.305373 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfvm2\" (UniqueName: \"kubernetes.io/projected/9011b3c2-ae6c-4252-bcce-594ffca54aef-kube-api-access-zfvm2\") pod \"9011b3c2-ae6c-4252-bcce-594ffca54aef\" (UID: \"9011b3c2-ae6c-4252-bcce-594ffca54aef\") " Sep 11 19:34:59.306222 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.306187 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-bundle" (OuterVolumeSpecName: "bundle") pod "9011b3c2-ae6c-4252-bcce-594ffca54aef" (UID: "9011b3c2-ae6c-4252-bcce-594ffca54aef"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:34:59.307963 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.307937 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9011b3c2-ae6c-4252-bcce-594ffca54aef-kube-api-access-zfvm2" (OuterVolumeSpecName: "kube-api-access-zfvm2") pod "9011b3c2-ae6c-4252-bcce-594ffca54aef" (UID: "9011b3c2-ae6c-4252-bcce-594ffca54aef"). InnerVolumeSpecName "kube-api-access-zfvm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:34:59.310605 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.310578 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-util" (OuterVolumeSpecName: "util") pod "9011b3c2-ae6c-4252-bcce-594ffca54aef" (UID: "9011b3c2-ae6c-4252-bcce-594ffca54aef"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:34:59.406692 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.406607 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-util\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:34:59.406692 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.406638 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9011b3c2-ae6c-4252-bcce-594ffca54aef-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:34:59.406692 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:34:59.406650 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfvm2\" (UniqueName: \"kubernetes.io/projected/9011b3c2-ae6c-4252-bcce-594ffca54aef-kube-api-access-zfvm2\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:00.087037 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:00.086997 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" event={"ID":"9011b3c2-ae6c-4252-bcce-594ffca54aef","Type":"ContainerDied","Data":"443197eaa019489185770a805af0ea14c0a3a3e3ae77eda9599d697a5446fc37"} Sep 11 19:35:00.087179 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:00.087060 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="443197eaa019489185770a805af0ea14c0a3a3e3ae77eda9599d697a5446fc37" Sep 11 19:35:00.087179 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:00.087019 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540rdfbz" Sep 11 19:35:06.196732 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.196701 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8"] Sep 11 19:35:06.197125 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.196952 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerName="util" Sep 11 19:35:06.197125 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.196962 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerName="util" Sep 11 19:35:06.197125 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.196974 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerName="extract" Sep 11 19:35:06.197125 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.196980 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerName="extract" Sep 11 19:35:06.197125 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.196992 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerName="pull" Sep 11 19:35:06.197125 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.196997 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerName="pull" Sep 11 19:35:06.197125 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.197042 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9011b3c2-ae6c-4252-bcce-594ffca54aef" containerName="extract" Sep 11 19:35:06.200116 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.200097 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" Sep 11 19:35:06.202791 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.202769 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 19:35:06.202921 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.202810 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-r2g85\"" Sep 11 19:35:06.202999 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.202981 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 19:35:06.214013 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.213991 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8"] Sep 11 19:35:06.255890 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.255853 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a924edc-0723-471c-aab1-1847791ef2dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-npxx8\" (UID: \"9a924edc-0723-471c-aab1-1847791ef2dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" Sep 11 19:35:06.256058 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.255899 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s46f\" (UniqueName: \"kubernetes.io/projected/9a924edc-0723-471c-aab1-1847791ef2dc-kube-api-access-8s46f\") pod \"cert-manager-operator-controller-manager-55c444bf86-npxx8\" (UID: \"9a924edc-0723-471c-aab1-1847791ef2dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" Sep 11 19:35:06.356908 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.356863 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a924edc-0723-471c-aab1-1847791ef2dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-npxx8\" (UID: \"9a924edc-0723-471c-aab1-1847791ef2dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" Sep 11 19:35:06.356908 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.356923 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s46f\" (UniqueName: \"kubernetes.io/projected/9a924edc-0723-471c-aab1-1847791ef2dc-kube-api-access-8s46f\") pod \"cert-manager-operator-controller-manager-55c444bf86-npxx8\" (UID: \"9a924edc-0723-471c-aab1-1847791ef2dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" Sep 11 19:35:06.357326 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.357301 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a924edc-0723-471c-aab1-1847791ef2dc-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-npxx8\" (UID: \"9a924edc-0723-471c-aab1-1847791ef2dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" Sep 11 19:35:06.368046 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.368018 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s46f\" (UniqueName: \"kubernetes.io/projected/9a924edc-0723-471c-aab1-1847791ef2dc-kube-api-access-8s46f\") pod \"cert-manager-operator-controller-manager-55c444bf86-npxx8\" (UID: \"9a924edc-0723-471c-aab1-1847791ef2dc\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" Sep 11 19:35:06.509161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.509109 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" Sep 11 19:35:06.789368 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:06.789346 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8"] Sep 11 19:35:06.791530 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:35:06.791491 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a924edc_0723_471c_aab1_1847791ef2dc.slice/crio-d6c9044ce058aa74f5605faa7d4abae54c98d958920d554944acf01e15a000a3 WatchSource:0}: Error finding container d6c9044ce058aa74f5605faa7d4abae54c98d958920d554944acf01e15a000a3: Status 404 returned error can't find the container with id d6c9044ce058aa74f5605faa7d4abae54c98d958920d554944acf01e15a000a3 Sep 11 19:35:07.113283 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:07.113194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" event={"ID":"9a924edc-0723-471c-aab1-1847791ef2dc","Type":"ContainerStarted","Data":"d6c9044ce058aa74f5605faa7d4abae54c98d958920d554944acf01e15a000a3"} Sep 11 19:35:09.123165 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:09.123075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" event={"ID":"9a924edc-0723-471c-aab1-1847791ef2dc","Type":"ContainerStarted","Data":"8adda14b014fb30c776d58bb0d4f6db36e37647d483090555203d2d9aa483936"} Sep 11 19:35:09.144328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:09.144280 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-npxx8" podStartSLOduration=1.157134988 podStartE2EDuration="3.144264046s" totalImagesPullingTime="1.987129058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:09.142281873 +0000 UTC m=+542.414789075" watchObservedRunningTime="2026-09-11 19:35:09.144264046 +0000 UTC m=+542.416771245" Sep 11 19:35:10.458677 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.458645 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg"] Sep 11 19:35:10.460819 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.460803 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.463405 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.463375 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 19:35:10.463555 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.463513 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p9dlz\"" Sep 11 19:35:10.464323 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.464308 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 19:35:10.474492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.474471 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg"] Sep 11 19:35:10.486757 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.486722 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpsbq\" (UniqueName: \"kubernetes.io/projected/a5d8860e-8fd9-4aaa-a97b-814b8c871639-kube-api-access-xpsbq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.486910 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.486890 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.486991 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.486974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.588226 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.588182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.588226 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.588231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpsbq\" (UniqueName: \"kubernetes.io/projected/a5d8860e-8fd9-4aaa-a97b-814b8c871639-kube-api-access-xpsbq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.588528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.588261 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.588617 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.588598 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.588678 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.588657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.597239 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.597221 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpsbq\" (UniqueName: \"kubernetes.io/projected/a5d8860e-8fd9-4aaa-a97b-814b8c871639-kube-api-access-xpsbq\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:10.770510 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:10.770472 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:11.044629 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:11.044521 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg"] Sep 11 19:35:11.046925 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:35:11.046899 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5d8860e_8fd9_4aaa_a97b_814b8c871639.slice/crio-215d5a0b8f1ec4765a44545ab2ff76373a7cbf666bc80e14616890375736b0e5 WatchSource:0}: Error finding container 215d5a0b8f1ec4765a44545ab2ff76373a7cbf666bc80e14616890375736b0e5: Status 404 returned error can't find the container with id 215d5a0b8f1ec4765a44545ab2ff76373a7cbf666bc80e14616890375736b0e5 Sep 11 19:35:11.132557 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:11.132523 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" event={"ID":"a5d8860e-8fd9-4aaa-a97b-814b8c871639","Type":"ContainerStarted","Data":"d590de311f2250c9cc7436fca532575cf888a42cddff8e65b59c216a0406ee5c"} Sep 11 19:35:11.132718 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:11.132571 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" event={"ID":"a5d8860e-8fd9-4aaa-a97b-814b8c871639","Type":"ContainerStarted","Data":"215d5a0b8f1ec4765a44545ab2ff76373a7cbf666bc80e14616890375736b0e5"} Sep 11 19:35:12.138648 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.138607 2559 generic.go:356] "Generic (PLEG): container finished" podID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerID="d590de311f2250c9cc7436fca532575cf888a42cddff8e65b59c216a0406ee5c" exitCode=0 Sep 11 19:35:12.138648 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.138648 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" event={"ID":"a5d8860e-8fd9-4aaa-a97b-814b8c871639","Type":"ContainerDied","Data":"d590de311f2250c9cc7436fca532575cf888a42cddff8e65b59c216a0406ee5c"} Sep 11 19:35:12.316948 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.316912 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v"] Sep 11 19:35:12.319068 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.319051 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" Sep 11 19:35:12.322097 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.322073 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 19:35:12.322210 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.322155 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-lhpc8\"" Sep 11 19:35:12.322255 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.322216 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 19:35:12.331536 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.331513 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v"] Sep 11 19:35:12.400788 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.400704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc5559b8-1428-4e99-8987-4a9953cd00c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m9p2v\" (UID: \"fc5559b8-1428-4e99-8987-4a9953cd00c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" Sep 11 19:35:12.400788 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.400742 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5q5\" (UniqueName: \"kubernetes.io/projected/fc5559b8-1428-4e99-8987-4a9953cd00c7-kube-api-access-zz5q5\") pod \"cert-manager-cainjector-559cdd8c87-m9p2v\" (UID: \"fc5559b8-1428-4e99-8987-4a9953cd00c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" Sep 11 19:35:12.501635 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.501588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc5559b8-1428-4e99-8987-4a9953cd00c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m9p2v\" (UID: \"fc5559b8-1428-4e99-8987-4a9953cd00c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" Sep 11 19:35:12.501807 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.501647 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5q5\" (UniqueName: \"kubernetes.io/projected/fc5559b8-1428-4e99-8987-4a9953cd00c7-kube-api-access-zz5q5\") pod \"cert-manager-cainjector-559cdd8c87-m9p2v\" (UID: \"fc5559b8-1428-4e99-8987-4a9953cd00c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" Sep 11 19:35:12.511998 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.511959 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fc5559b8-1428-4e99-8987-4a9953cd00c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-m9p2v\" (UID: \"fc5559b8-1428-4e99-8987-4a9953cd00c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" Sep 11 19:35:12.512117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.512095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5q5\" (UniqueName: \"kubernetes.io/projected/fc5559b8-1428-4e99-8987-4a9953cd00c7-kube-api-access-zz5q5\") pod \"cert-manager-cainjector-559cdd8c87-m9p2v\" (UID: \"fc5559b8-1428-4e99-8987-4a9953cd00c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" Sep 11 19:35:12.628110 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.628076 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" Sep 11 19:35:12.897409 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:12.897385 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v"] Sep 11 19:35:12.899769 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:35:12.899739 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc5559b8_1428_4e99_8987_4a9953cd00c7.slice/crio-fac44b663ab56a08fbcb580862b27fc944be66f7eae8495ed6316a02f08b57e5 WatchSource:0}: Error finding container fac44b663ab56a08fbcb580862b27fc944be66f7eae8495ed6316a02f08b57e5: Status 404 returned error can't find the container with id fac44b663ab56a08fbcb580862b27fc944be66f7eae8495ed6316a02f08b57e5 Sep 11 19:35:13.144855 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:13.144814 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" event={"ID":"fc5559b8-1428-4e99-8987-4a9953cd00c7","Type":"ContainerStarted","Data":"fac44b663ab56a08fbcb580862b27fc944be66f7eae8495ed6316a02f08b57e5"} Sep 11 19:35:15.157385 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:15.157346 2559 generic.go:356] "Generic (PLEG): container finished" podID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerID="c1ea2f8bc8cd5dcfbbd237c3a90a7fb8ef41c5b3faac2629c20842da04269722" exitCode=0 Sep 11 19:35:15.157979 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:15.157398 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" event={"ID":"a5d8860e-8fd9-4aaa-a97b-814b8c871639","Type":"ContainerDied","Data":"c1ea2f8bc8cd5dcfbbd237c3a90a7fb8ef41c5b3faac2629c20842da04269722"} Sep 11 19:35:16.163398 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:16.163356 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" event={"ID":"fc5559b8-1428-4e99-8987-4a9953cd00c7","Type":"ContainerStarted","Data":"54818514a25f9f784b0e4052486762ad72676e818af5acc8225faedf88f0a272"} Sep 11 19:35:16.165426 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:16.165401 2559 generic.go:356] "Generic (PLEG): container finished" podID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerID="aa65ae2042837fbecc75e2641cafbf60bc9571446411a49d17e30cd186ec2ea5" exitCode=0 Sep 11 19:35:16.165560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:16.165459 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" event={"ID":"a5d8860e-8fd9-4aaa-a97b-814b8c871639","Type":"ContainerDied","Data":"aa65ae2042837fbecc75e2641cafbf60bc9571446411a49d17e30cd186ec2ea5"} Sep 11 19:35:16.179413 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:16.179370 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-m9p2v" podStartSLOduration=1.529522574 podStartE2EDuration="4.179354758s" totalImagesPullingTime="2.649832184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:16.177850186 +0000 UTC m=+549.450357369" watchObservedRunningTime="2026-09-11 19:35:16.179354758 +0000 UTC m=+549.451861958" Sep 11 19:35:17.290499 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.290476 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:17.344165 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.344130 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-bundle\") pod \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " Sep 11 19:35:17.344165 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.344169 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpsbq\" (UniqueName: \"kubernetes.io/projected/a5d8860e-8fd9-4aaa-a97b-814b8c871639-kube-api-access-xpsbq\") pod \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " Sep 11 19:35:17.344403 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.344199 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-util\") pod \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\" (UID: \"a5d8860e-8fd9-4aaa-a97b-814b8c871639\") " Sep 11 19:35:17.344633 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.344607 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-bundle" (OuterVolumeSpecName: "bundle") pod "a5d8860e-8fd9-4aaa-a97b-814b8c871639" (UID: "a5d8860e-8fd9-4aaa-a97b-814b8c871639"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:35:17.346510 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.346480 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5d8860e-8fd9-4aaa-a97b-814b8c871639-kube-api-access-xpsbq" (OuterVolumeSpecName: "kube-api-access-xpsbq") pod "a5d8860e-8fd9-4aaa-a97b-814b8c871639" (UID: "a5d8860e-8fd9-4aaa-a97b-814b8c871639"). InnerVolumeSpecName "kube-api-access-xpsbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:35:17.348465 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.348422 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-util" (OuterVolumeSpecName: "util") pod "a5d8860e-8fd9-4aaa-a97b-814b8c871639" (UID: "a5d8860e-8fd9-4aaa-a97b-814b8c871639"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:35:17.445634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.445548 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-util\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:17.445634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.445575 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a5d8860e-8fd9-4aaa-a97b-814b8c871639-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:17.445634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:17.445585 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpsbq\" (UniqueName: \"kubernetes.io/projected/a5d8860e-8fd9-4aaa-a97b-814b8c871639-kube-api-access-xpsbq\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:18.175999 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:18.175958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" event={"ID":"a5d8860e-8fd9-4aaa-a97b-814b8c871639","Type":"ContainerDied","Data":"215d5a0b8f1ec4765a44545ab2ff76373a7cbf666bc80e14616890375736b0e5"} Sep 11 19:35:18.175999 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:18.175989 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f667zg" Sep 11 19:35:18.176190 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:18.176031 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="215d5a0b8f1ec4765a44545ab2ff76373a7cbf666bc80e14616890375736b0e5" Sep 11 19:35:21.247534 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.247494 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qvqgb"] Sep 11 19:35:21.247928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.247766 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerName="util" Sep 11 19:35:21.247928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.247777 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerName="util" Sep 11 19:35:21.247928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.247796 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerName="extract" Sep 11 19:35:21.247928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.247801 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerName="extract" Sep 11 19:35:21.247928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.247809 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerName="pull" Sep 11 19:35:21.247928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.247815 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerName="pull" Sep 11 19:35:21.247928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.247856 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5d8860e-8fd9-4aaa-a97b-814b8c871639" containerName="extract" Sep 11 19:35:21.251690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.251668 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qvqgb" Sep 11 19:35:21.254326 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.254309 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-27p9n\"" Sep 11 19:35:21.260470 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.260424 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qvqgb"] Sep 11 19:35:21.375783 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.375743 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7987f2b5-21d0-40b6-bcde-4fb33160377b-bound-sa-token\") pod \"cert-manager-55b4d954d5-qvqgb\" (UID: \"7987f2b5-21d0-40b6-bcde-4fb33160377b\") " pod="cert-manager/cert-manager-55b4d954d5-qvqgb" Sep 11 19:35:21.375944 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.375805 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rblwn\" (UniqueName: \"kubernetes.io/projected/7987f2b5-21d0-40b6-bcde-4fb33160377b-kube-api-access-rblwn\") pod \"cert-manager-55b4d954d5-qvqgb\" (UID: \"7987f2b5-21d0-40b6-bcde-4fb33160377b\") " pod="cert-manager/cert-manager-55b4d954d5-qvqgb" Sep 11 19:35:21.476341 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.476310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rblwn\" (UniqueName: \"kubernetes.io/projected/7987f2b5-21d0-40b6-bcde-4fb33160377b-kube-api-access-rblwn\") pod \"cert-manager-55b4d954d5-qvqgb\" (UID: \"7987f2b5-21d0-40b6-bcde-4fb33160377b\") " pod="cert-manager/cert-manager-55b4d954d5-qvqgb" Sep 11 19:35:21.476556 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.476368 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7987f2b5-21d0-40b6-bcde-4fb33160377b-bound-sa-token\") pod \"cert-manager-55b4d954d5-qvqgb\" (UID: \"7987f2b5-21d0-40b6-bcde-4fb33160377b\") " pod="cert-manager/cert-manager-55b4d954d5-qvqgb" Sep 11 19:35:21.485586 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.485559 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rblwn\" (UniqueName: \"kubernetes.io/projected/7987f2b5-21d0-40b6-bcde-4fb33160377b-kube-api-access-rblwn\") pod \"cert-manager-55b4d954d5-qvqgb\" (UID: \"7987f2b5-21d0-40b6-bcde-4fb33160377b\") " pod="cert-manager/cert-manager-55b4d954d5-qvqgb" Sep 11 19:35:21.485724 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.485601 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7987f2b5-21d0-40b6-bcde-4fb33160377b-bound-sa-token\") pod \"cert-manager-55b4d954d5-qvqgb\" (UID: \"7987f2b5-21d0-40b6-bcde-4fb33160377b\") " pod="cert-manager/cert-manager-55b4d954d5-qvqgb" Sep 11 19:35:21.561096 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.561002 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-qvqgb" Sep 11 19:35:21.831636 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:21.831552 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-qvqgb"] Sep 11 19:35:21.834576 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:35:21.834550 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7987f2b5_21d0_40b6_bcde_4fb33160377b.slice/crio-ccd22ff75cd4ee6b4827ebafaf103bf67b9a9ba24f88823e96ef63857865daa7 WatchSource:0}: Error finding container ccd22ff75cd4ee6b4827ebafaf103bf67b9a9ba24f88823e96ef63857865daa7: Status 404 returned error can't find the container with id ccd22ff75cd4ee6b4827ebafaf103bf67b9a9ba24f88823e96ef63857865daa7 Sep 11 19:35:22.195186 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:22.195092 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qvqgb" event={"ID":"7987f2b5-21d0-40b6-bcde-4fb33160377b","Type":"ContainerStarted","Data":"831e9b47c42c2125e4ed20f72ecc0b74573c95ed5b53e034b3c0c18721a0f1e4"} Sep 11 19:35:22.195186 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:22.195147 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-qvqgb" event={"ID":"7987f2b5-21d0-40b6-bcde-4fb33160377b","Type":"ContainerStarted","Data":"ccd22ff75cd4ee6b4827ebafaf103bf67b9a9ba24f88823e96ef63857865daa7"} Sep 11 19:35:22.210948 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:22.210904 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-qvqgb" podStartSLOduration=1.2108909429999999 podStartE2EDuration="1.210890943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:22.209636531 +0000 UTC m=+555.482143732" watchObservedRunningTime="2026-09-11 19:35:22.210890943 +0000 UTC m=+555.483398143" Sep 11 19:35:30.933749 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:30.933707 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2"] Sep 11 19:35:30.936145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:30.936122 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:30.938788 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:30.938764 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 19:35:30.938898 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:30.938787 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 19:35:30.939866 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:30.939840 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p9dlz\"" Sep 11 19:35:30.947010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:30.946990 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2"] Sep 11 19:35:31.051834 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.051801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.052031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.051848 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.052031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.051871 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnnmn\" (UniqueName: \"kubernetes.io/projected/125d8509-7ab8-4a11-85dc-409ce7796860-kube-api-access-fnnmn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.152415 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.152376 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.152616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.152426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.152616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.152471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnnmn\" (UniqueName: \"kubernetes.io/projected/125d8509-7ab8-4a11-85dc-409ce7796860-kube-api-access-fnnmn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.152806 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.152775 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.152884 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.152857 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.161505 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.161481 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnnmn\" (UniqueName: \"kubernetes.io/projected/125d8509-7ab8-4a11-85dc-409ce7796860-kube-api-access-fnnmn\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.244958 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.244933 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:31.514311 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:31.514229 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2"] Sep 11 19:35:31.517012 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:35:31.516985 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125d8509_7ab8_4a11_85dc_409ce7796860.slice/crio-5f22cc94c4726db61982d33f9a8e68a66bd6220864073d3324dedda17df5829a WatchSource:0}: Error finding container 5f22cc94c4726db61982d33f9a8e68a66bd6220864073d3324dedda17df5829a: Status 404 returned error can't find the container with id 5f22cc94c4726db61982d33f9a8e68a66bd6220864073d3324dedda17df5829a Sep 11 19:35:32.240560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:32.240526 2559 generic.go:356] "Generic (PLEG): container finished" podID="125d8509-7ab8-4a11-85dc-409ce7796860" containerID="e8c2a4b1d22d58cdf845e97d9a109558e53eef77944b8e5722031446be1d90e3" exitCode=0 Sep 11 19:35:32.240935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:32.240581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" event={"ID":"125d8509-7ab8-4a11-85dc-409ce7796860","Type":"ContainerDied","Data":"e8c2a4b1d22d58cdf845e97d9a109558e53eef77944b8e5722031446be1d90e3"} Sep 11 19:35:32.240935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:32.240623 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" event={"ID":"125d8509-7ab8-4a11-85dc-409ce7796860","Type":"ContainerStarted","Data":"5f22cc94c4726db61982d33f9a8e68a66bd6220864073d3324dedda17df5829a"} Sep 11 19:35:33.246750 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:33.246716 2559 generic.go:356] "Generic (PLEG): container finished" podID="125d8509-7ab8-4a11-85dc-409ce7796860" containerID="9bf7b810641bfc2f40339923c90052e173eada6e71fe021979432fc8a318fe44" exitCode=0 Sep 11 19:35:33.247132 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:33.246798 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" event={"ID":"125d8509-7ab8-4a11-85dc-409ce7796860","Type":"ContainerDied","Data":"9bf7b810641bfc2f40339923c90052e173eada6e71fe021979432fc8a318fe44"} Sep 11 19:35:34.253266 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:34.253233 2559 generic.go:356] "Generic (PLEG): container finished" podID="125d8509-7ab8-4a11-85dc-409ce7796860" containerID="94acc0c38e2d2ae5ef3d5a2f322883679a749e3d515ad89ec07b0711f46c5840" exitCode=0 Sep 11 19:35:34.253671 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:34.253303 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" event={"ID":"125d8509-7ab8-4a11-85dc-409ce7796860","Type":"ContainerDied","Data":"94acc0c38e2d2ae5ef3d5a2f322883679a749e3d515ad89ec07b0711f46c5840"} Sep 11 19:35:35.382643 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.382601 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:35.488306 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.488264 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-util\") pod \"125d8509-7ab8-4a11-85dc-409ce7796860\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " Sep 11 19:35:35.488523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.488318 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnnmn\" (UniqueName: \"kubernetes.io/projected/125d8509-7ab8-4a11-85dc-409ce7796860-kube-api-access-fnnmn\") pod \"125d8509-7ab8-4a11-85dc-409ce7796860\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " Sep 11 19:35:35.488523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.488365 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-bundle\") pod \"125d8509-7ab8-4a11-85dc-409ce7796860\" (UID: \"125d8509-7ab8-4a11-85dc-409ce7796860\") " Sep 11 19:35:35.489124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.489097 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-bundle" (OuterVolumeSpecName: "bundle") pod "125d8509-7ab8-4a11-85dc-409ce7796860" (UID: "125d8509-7ab8-4a11-85dc-409ce7796860"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:35:35.490636 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.490608 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125d8509-7ab8-4a11-85dc-409ce7796860-kube-api-access-fnnmn" (OuterVolumeSpecName: "kube-api-access-fnnmn") pod "125d8509-7ab8-4a11-85dc-409ce7796860" (UID: "125d8509-7ab8-4a11-85dc-409ce7796860"). InnerVolumeSpecName "kube-api-access-fnnmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:35:35.494177 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.494147 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-util" (OuterVolumeSpecName: "util") pod "125d8509-7ab8-4a11-85dc-409ce7796860" (UID: "125d8509-7ab8-4a11-85dc-409ce7796860"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:35:35.589151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.589048 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-util\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:35.589151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.589093 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnnmn\" (UniqueName: \"kubernetes.io/projected/125d8509-7ab8-4a11-85dc-409ce7796860-kube-api-access-fnnmn\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:35.589151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:35.589103 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/125d8509-7ab8-4a11-85dc-409ce7796860-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:36.264248 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:36.264215 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" Sep 11 19:35:36.264248 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:36.264230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afsm6q2" event={"ID":"125d8509-7ab8-4a11-85dc-409ce7796860","Type":"ContainerDied","Data":"5f22cc94c4726db61982d33f9a8e68a66bd6220864073d3324dedda17df5829a"} Sep 11 19:35:36.264415 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:36.264276 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f22cc94c4726db61982d33f9a8e68a66bd6220864073d3324dedda17df5829a" Sep 11 19:35:42.284942 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.284907 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j"] Sep 11 19:35:42.285398 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.285205 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125d8509-7ab8-4a11-85dc-409ce7796860" containerName="extract" Sep 11 19:35:42.285398 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.285218 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="125d8509-7ab8-4a11-85dc-409ce7796860" containerName="extract" Sep 11 19:35:42.285398 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.285229 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125d8509-7ab8-4a11-85dc-409ce7796860" containerName="pull" Sep 11 19:35:42.285398 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.285234 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="125d8509-7ab8-4a11-85dc-409ce7796860" containerName="pull" Sep 11 19:35:42.285398 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.285258 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="125d8509-7ab8-4a11-85dc-409ce7796860" containerName="util" Sep 11 19:35:42.285398 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.285267 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="125d8509-7ab8-4a11-85dc-409ce7796860" containerName="util" Sep 11 19:35:42.285398 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.285315 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="125d8509-7ab8-4a11-85dc-409ce7796860" containerName="extract" Sep 11 19:35:42.287933 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.287914 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.290543 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.290521 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 19:35:42.290655 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.290521 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 19:35:42.291543 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.291525 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p9dlz\"" Sep 11 19:35:42.297144 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.297124 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j"] Sep 11 19:35:42.449684 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.449647 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.449854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.449732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.449854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.449761 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj4nj\" (UniqueName: \"kubernetes.io/projected/8e43d78c-4860-4d72-8df8-0f03cb43dc08-kube-api-access-qj4nj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.550316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.550212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.550316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.550307 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.550531 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.550336 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj4nj\" (UniqueName: \"kubernetes.io/projected/8e43d78c-4860-4d72-8df8-0f03cb43dc08-kube-api-access-qj4nj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.550653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.550633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.550693 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.550654 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.558391 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.558363 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj4nj\" (UniqueName: \"kubernetes.io/projected/8e43d78c-4860-4d72-8df8-0f03cb43dc08-kube-api-access-qj4nj\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.599530 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.599502 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:42.874530 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:42.874508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j"] Sep 11 19:35:42.877226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:35:42.877196 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e43d78c_4860_4d72_8df8_0f03cb43dc08.slice/crio-19e0898350178f0158014bc6f4ad73bd9640ccb1adf06a9b444156f7045488a8 WatchSource:0}: Error finding container 19e0898350178f0158014bc6f4ad73bd9640ccb1adf06a9b444156f7045488a8: Status 404 returned error can't find the container with id 19e0898350178f0158014bc6f4ad73bd9640ccb1adf06a9b444156f7045488a8 Sep 11 19:35:43.295177 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:43.295137 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerID="730776a65a2b8e1fb684d255b0a06e4d5ca5fec70bcdadd67a5f1540074be6ce" exitCode=0 Sep 11 19:35:43.295587 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:43.295226 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" event={"ID":"8e43d78c-4860-4d72-8df8-0f03cb43dc08","Type":"ContainerDied","Data":"730776a65a2b8e1fb684d255b0a06e4d5ca5fec70bcdadd67a5f1540074be6ce"} Sep 11 19:35:43.295587 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:43.295278 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" event={"ID":"8e43d78c-4860-4d72-8df8-0f03cb43dc08","Type":"ContainerStarted","Data":"19e0898350178f0158014bc6f4ad73bd9640ccb1adf06a9b444156f7045488a8"} Sep 11 19:35:44.301905 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.301877 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerID="9d05145434b11637d82bba0b12d1534e863d52c261aaf1bd1d345f16b6308cb5" exitCode=0 Sep 11 19:35:44.302276 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.301965 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" event={"ID":"8e43d78c-4860-4d72-8df8-0f03cb43dc08","Type":"ContainerDied","Data":"9d05145434b11637d82bba0b12d1534e863d52c261aaf1bd1d345f16b6308cb5"} Sep 11 19:35:44.338173 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.338147 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj"] Sep 11 19:35:44.342645 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.342626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.345408 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.345392 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 19:35:44.345555 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.345394 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 19:35:44.345731 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.345710 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 19:35:44.346003 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.345984 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2rjxh\"" Sep 11 19:35:44.346090 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.346003 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 19:35:44.358145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.358120 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj"] Sep 11 19:35:44.465772 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.465748 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcb4\" (UniqueName: \"kubernetes.io/projected/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-kube-api-access-4dcb4\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.465901 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.465789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.465901 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.465846 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-tmp\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.465901 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.465877 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-manifests\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.466028 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.465902 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.566383 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.566301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcb4\" (UniqueName: \"kubernetes.io/projected/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-kube-api-access-4dcb4\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.566383 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.566339 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.566383 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.566364 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-tmp\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.566664 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.566390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-manifests\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.566664 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.566409 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.566843 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.566820 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-manifests\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.568949 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.568925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.569056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.568982 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-tmp\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.569223 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.569200 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.579957 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.579930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcb4\" (UniqueName: \"kubernetes.io/projected/d4bffeba-f3e3-49a1-95c6-31b4a31d0755-kube-api-access-4dcb4\") pod \"opendatahub-operator-controller-manager-7c47b89ccd-kgkdj\" (UID: \"d4bffeba-f3e3-49a1-95c6-31b4a31d0755\") " pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.653071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.653036 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:44.934432 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:44.934406 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj"] Sep 11 19:35:44.936293 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:35:44.936266 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4bffeba_f3e3_49a1_95c6_31b4a31d0755.slice/crio-843322de7553efb3599fb91f50af427c8fbbc69f8a05a8a793f53e8f58948c84 WatchSource:0}: Error finding container 843322de7553efb3599fb91f50af427c8fbbc69f8a05a8a793f53e8f58948c84: Status 404 returned error can't find the container with id 843322de7553efb3599fb91f50af427c8fbbc69f8a05a8a793f53e8f58948c84 Sep 11 19:35:45.307752 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:45.307715 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" event={"ID":"d4bffeba-f3e3-49a1-95c6-31b4a31d0755","Type":"ContainerStarted","Data":"843322de7553efb3599fb91f50af427c8fbbc69f8a05a8a793f53e8f58948c84"} Sep 11 19:35:45.309792 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:45.309769 2559 generic.go:356] "Generic (PLEG): container finished" podID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerID="636c99a9d6e8f34c4782e0f9fddb8eedcb5451808d1c68094a89dfb96ccad9c4" exitCode=0 Sep 11 19:35:45.309907 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:45.309815 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" event={"ID":"8e43d78c-4860-4d72-8df8-0f03cb43dc08","Type":"ContainerDied","Data":"636c99a9d6e8f34c4782e0f9fddb8eedcb5451808d1c68094a89dfb96ccad9c4"} Sep 11 19:35:47.943548 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:47.943518 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:48.100420 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.100388 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-util\") pod \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " Sep 11 19:35:48.100628 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.100467 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-bundle\") pod \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " Sep 11 19:35:48.100628 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.100507 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj4nj\" (UniqueName: \"kubernetes.io/projected/8e43d78c-4860-4d72-8df8-0f03cb43dc08-kube-api-access-qj4nj\") pod \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\" (UID: \"8e43d78c-4860-4d72-8df8-0f03cb43dc08\") " Sep 11 19:35:48.101228 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.101146 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-bundle" (OuterVolumeSpecName: "bundle") pod "8e43d78c-4860-4d72-8df8-0f03cb43dc08" (UID: "8e43d78c-4860-4d72-8df8-0f03cb43dc08"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:35:48.102819 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.102790 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e43d78c-4860-4d72-8df8-0f03cb43dc08-kube-api-access-qj4nj" (OuterVolumeSpecName: "kube-api-access-qj4nj") pod "8e43d78c-4860-4d72-8df8-0f03cb43dc08" (UID: "8e43d78c-4860-4d72-8df8-0f03cb43dc08"). InnerVolumeSpecName "kube-api-access-qj4nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:35:48.105851 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.105828 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-util" (OuterVolumeSpecName: "util") pod "8e43d78c-4860-4d72-8df8-0f03cb43dc08" (UID: "8e43d78c-4860-4d72-8df8-0f03cb43dc08"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:35:48.201404 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.201326 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj4nj\" (UniqueName: \"kubernetes.io/projected/8e43d78c-4860-4d72-8df8-0f03cb43dc08-kube-api-access-qj4nj\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:48.201404 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.201359 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-util\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:48.201404 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.201369 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8e43d78c-4860-4d72-8df8-0f03cb43dc08-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:35:48.326503 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.326464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" event={"ID":"d4bffeba-f3e3-49a1-95c6-31b4a31d0755","Type":"ContainerStarted","Data":"141d00067fc28ffa5e2a361750dde4415056f481c056dc30f2c240e2674ca90f"} Sep 11 19:35:48.328218 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.328193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" event={"ID":"8e43d78c-4860-4d72-8df8-0f03cb43dc08","Type":"ContainerDied","Data":"19e0898350178f0158014bc6f4ad73bd9640ccb1adf06a9b444156f7045488a8"} Sep 11 19:35:48.328335 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.328228 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19e0898350178f0158014bc6f4ad73bd9640ccb1adf06a9b444156f7045488a8" Sep 11 19:35:48.328335 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:48.328241 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b07gv8j" Sep 11 19:35:49.334790 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:49.334761 2559 generic.go:356] "Generic (PLEG): container finished" podID="d4bffeba-f3e3-49a1-95c6-31b4a31d0755" containerID="141d00067fc28ffa5e2a361750dde4415056f481c056dc30f2c240e2674ca90f" exitCode=0 Sep 11 19:35:49.335159 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:49.334849 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" event={"ID":"d4bffeba-f3e3-49a1-95c6-31b4a31d0755","Type":"ContainerDied","Data":"141d00067fc28ffa5e2a361750dde4415056f481c056dc30f2c240e2674ca90f"} Sep 11 19:35:50.341247 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:50.341203 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" event={"ID":"d4bffeba-f3e3-49a1-95c6-31b4a31d0755","Type":"ContainerStarted","Data":"252540d08ecf0701094d43bc4d3cad8a4d9e4801c202943ccbc35cd57dcfc808"} Sep 11 19:35:50.341728 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:50.341317 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:35:50.372493 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:35:50.372420 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" podStartSLOduration=1.944099612 podStartE2EDuration="6.372403238s" totalImagesPullingTime="3.364320083s" totalInitContainerRuntime="1.063983543s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:35:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:35:50.369803876 +0000 UTC m=+583.642311057" watchObservedRunningTime="2026-09-11 19:35:50.372403238 +0000 UTC m=+583.644910439" Sep 11 19:36:01.348380 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:01.348341 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:36:09.044883 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.044846 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-596df76fbd-kls4w"] Sep 11 19:36:09.045533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.045122 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerName="pull" Sep 11 19:36:09.045533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.045132 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerName="pull" Sep 11 19:36:09.045533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.045152 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerName="extract" Sep 11 19:36:09.045533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.045157 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerName="extract" Sep 11 19:36:09.045533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.045166 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerName="util" Sep 11 19:36:09.045533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.045171 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerName="util" Sep 11 19:36:09.045533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.045215 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e43d78c-4860-4d72-8df8-0f03cb43dc08" containerName="extract" Sep 11 19:36:09.051793 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.051761 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.054757 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.054734 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cz7vf\"" Sep 11 19:36:09.054880 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.054793 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 19:36:09.055896 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.055873 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 19:36:09.056016 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.055913 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 19:36:09.056016 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.055951 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 19:36:09.062597 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.062569 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-596df76fbd-kls4w"] Sep 11 19:36:09.070369 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.070341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112e02c3-a598-4433-b801-d748a0a949f9-tmp\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.070508 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.070385 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112e02c3-a598-4433-b801-d748a0a949f9-tls-certs\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.070508 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.070416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zqc\" (UniqueName: \"kubernetes.io/projected/112e02c3-a598-4433-b801-d748a0a949f9-kube-api-access-46zqc\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.171760 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.171724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112e02c3-a598-4433-b801-d748a0a949f9-tmp\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.171760 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.171761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112e02c3-a598-4433-b801-d748a0a949f9-tls-certs\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.171958 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.171782 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zqc\" (UniqueName: \"kubernetes.io/projected/112e02c3-a598-4433-b801-d748a0a949f9-kube-api-access-46zqc\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.171958 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:36:09.171875 2559 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 11 19:36:09.171958 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:36:09.171952 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/112e02c3-a598-4433-b801-d748a0a949f9-tls-certs podName:112e02c3-a598-4433-b801-d748a0a949f9 nodeName:}" failed. No retries permitted until 2026-09-11 19:36:09.671935803 +0000 UTC m=+602.944442982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/112e02c3-a598-4433-b801-d748a0a949f9-tls-certs") pod "kube-auth-proxy-596df76fbd-kls4w" (UID: "112e02c3-a598-4433-b801-d748a0a949f9") : secret "kube-auth-proxy-tls" not found Sep 11 19:36:09.174266 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.174243 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/112e02c3-a598-4433-b801-d748a0a949f9-tmp\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.180264 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.180241 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zqc\" (UniqueName: \"kubernetes.io/projected/112e02c3-a598-4433-b801-d748a0a949f9-kube-api-access-46zqc\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.676545 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.676497 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112e02c3-a598-4433-b801-d748a0a949f9-tls-certs\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.679061 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.679034 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/112e02c3-a598-4433-b801-d748a0a949f9-tls-certs\") pod \"kube-auth-proxy-596df76fbd-kls4w\" (UID: \"112e02c3-a598-4433-b801-d748a0a949f9\") " pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:09.964875 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:09.964799 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" Sep 11 19:36:10.165750 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.165719 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r"] Sep 11 19:36:10.167885 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.167868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.170786 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.170764 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 19:36:10.170911 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.170785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 19:36:10.170911 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.170785 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 19:36:10.170911 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.170767 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 19:36:10.171563 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.171548 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dmm5l\"" Sep 11 19:36:10.185112 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.185080 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r"] Sep 11 19:36:10.236209 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.236187 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-596df76fbd-kls4w"] Sep 11 19:36:10.238233 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:36:10.238207 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod112e02c3_a598_4433_b801_d748a0a949f9.slice/crio-d2b175d94ecb6295225143b8f8990fe932b440c6acb35cbebca168cf5ad6d255 WatchSource:0}: Error finding container d2b175d94ecb6295225143b8f8990fe932b440c6acb35cbebca168cf5ad6d255: Status 404 returned error can't find the container with id d2b175d94ecb6295225143b8f8990fe932b440c6acb35cbebca168cf5ad6d255 Sep 11 19:36:10.281641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.281609 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005e6b93-4a54-420f-bc2f-99c9525af30f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.281778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.281659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.281778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.281685 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx572\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-kube-api-access-xx572\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.281778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.281734 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.281778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.281763 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.281962 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.281811 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.281962 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.281856 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.382492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.382426 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.382492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.382491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.382730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.382525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005e6b93-4a54-420f-bc2f-99c9525af30f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.382730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.382549 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.382730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.382568 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx572\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-kube-api-access-xx572\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.382730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.382602 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.382730 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.382632 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.383210 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.383185 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.384991 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.384970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005e6b93-4a54-420f-bc2f-99c9525af30f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.385133 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.385113 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.385421 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.385402 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.385502 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.385412 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.401047 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.401016 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.403803 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.403783 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx572\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-kube-api-access-xx572\") pod \"istiod-openshift-gateway-7b7995d5cd-56f5r\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.431175 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.431143 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" event={"ID":"112e02c3-a598-4433-b801-d748a0a949f9","Type":"ContainerStarted","Data":"d2b175d94ecb6295225143b8f8990fe932b440c6acb35cbebca168cf5ad6d255"} Sep 11 19:36:10.477660 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.477626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:10.756935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:10.756903 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r"] Sep 11 19:36:10.759858 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:36:10.759831 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005e6b93_4a54_420f_bc2f_99c9525af30f.slice/crio-ad114b8601ecbda300afdfdc7032e807680c9dd34d6ed1d373ce9e55c1aed05b WatchSource:0}: Error finding container ad114b8601ecbda300afdfdc7032e807680c9dd34d6ed1d373ce9e55c1aed05b: Status 404 returned error can't find the container with id ad114b8601ecbda300afdfdc7032e807680c9dd34d6ed1d373ce9e55c1aed05b Sep 11 19:36:11.439472 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:11.439407 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" event={"ID":"005e6b93-4a54-420f-bc2f-99c9525af30f","Type":"ContainerStarted","Data":"ad114b8601ecbda300afdfdc7032e807680c9dd34d6ed1d373ce9e55c1aed05b"} Sep 11 19:36:14.313995 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:14.313951 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 19:36:14.314358 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:14.314033 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"} Sep 11 19:36:14.462290 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:14.462249 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" event={"ID":"112e02c3-a598-4433-b801-d748a0a949f9","Type":"ContainerStarted","Data":"dc6a2a0f723d01f03f6e13e242d90a4d18fef36621ef96c17df9ad89148c2aec"} Sep 11 19:36:14.464270 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:14.464241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" event={"ID":"005e6b93-4a54-420f-bc2f-99c9525af30f","Type":"ContainerStarted","Data":"7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0"} Sep 11 19:36:14.464428 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:14.464375 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:14.483063 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:14.483008 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-596df76fbd-kls4w" podStartSLOduration=1.404034904 podStartE2EDuration="5.482992121s" totalImagesPullingTime="4.078957217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:36:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:36:14.480667032 +0000 UTC m=+607.753174231" watchObservedRunningTime="2026-09-11 19:36:14.482992121 +0000 UTC m=+607.755499353" Sep 11 19:36:14.505424 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:14.505170 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" podStartSLOduration=0.953327649 podStartE2EDuration="4.505145693s" totalImagesPullingTime="3.551818044s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:36:14.502900298 +0000 UTC m=+607.775407498" watchObservedRunningTime="2026-09-11 19:36:14.505145693 +0000 UTC m=+607.777652894" Sep 11 19:36:15.472568 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:15.472538 2559 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-56f5r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 19:36:15.472988 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:15.472591 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" podUID="005e6b93-4a54-420f-bc2f-99c9525af30f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 19:36:18.472207 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:18.472170 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:24.131936 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.131899 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh"] Sep 11 19:36:24.136341 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.136322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:24.139885 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.139853 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bndsk\"" Sep 11 19:36:24.140352 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.140327 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 19:36:24.150284 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.150262 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh"] Sep 11 19:36:24.213559 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.213525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e2c2640-d759-41bc-bc41-7f57b39cb976-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-88bmh\" (UID: \"1e2c2640-d759-41bc-bc41-7f57b39cb976\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:24.213786 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.213680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzfxm\" (UniqueName: \"kubernetes.io/projected/1e2c2640-d759-41bc-bc41-7f57b39cb976-kube-api-access-mzfxm\") pod \"llmisvc-controller-manager-777c6bd7d5-88bmh\" (UID: \"1e2c2640-d759-41bc-bc41-7f57b39cb976\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:24.314949 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.314912 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzfxm\" (UniqueName: \"kubernetes.io/projected/1e2c2640-d759-41bc-bc41-7f57b39cb976-kube-api-access-mzfxm\") pod \"llmisvc-controller-manager-777c6bd7d5-88bmh\" (UID: \"1e2c2640-d759-41bc-bc41-7f57b39cb976\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:24.315134 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.314956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e2c2640-d759-41bc-bc41-7f57b39cb976-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-88bmh\" (UID: \"1e2c2640-d759-41bc-bc41-7f57b39cb976\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:24.315134 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:36:24.315083 2559 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Sep 11 19:36:24.315215 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:36:24.315146 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e2c2640-d759-41bc-bc41-7f57b39cb976-cert podName:1e2c2640-d759-41bc-bc41-7f57b39cb976 nodeName:}" failed. No retries permitted until 2026-09-11 19:36:24.81512921 +0000 UTC m=+618.087636388 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1e2c2640-d759-41bc-bc41-7f57b39cb976-cert") pod "llmisvc-controller-manager-777c6bd7d5-88bmh" (UID: "1e2c2640-d759-41bc-bc41-7f57b39cb976") : secret "llmisvc-webhook-server-cert" not found Sep 11 19:36:24.332979 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.332943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzfxm\" (UniqueName: \"kubernetes.io/projected/1e2c2640-d759-41bc-bc41-7f57b39cb976-kube-api-access-mzfxm\") pod \"llmisvc-controller-manager-777c6bd7d5-88bmh\" (UID: \"1e2c2640-d759-41bc-bc41-7f57b39cb976\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:24.819353 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.819309 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e2c2640-d759-41bc-bc41-7f57b39cb976-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-88bmh\" (UID: \"1e2c2640-d759-41bc-bc41-7f57b39cb976\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:24.821856 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:24.821827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1e2c2640-d759-41bc-bc41-7f57b39cb976-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-88bmh\" (UID: \"1e2c2640-d759-41bc-bc41-7f57b39cb976\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:25.046097 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:25.046063 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:25.330883 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:25.330855 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh"] Sep 11 19:36:25.333772 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:36:25.333740 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1e2c2640_d759_41bc_bc41_7f57b39cb976.slice/crio-cc257cc0efbec4ccb1b6b93a1747a6d14c071b2b304234ef8c53fc822f9a6004 WatchSource:0}: Error finding container cc257cc0efbec4ccb1b6b93a1747a6d14c071b2b304234ef8c53fc822f9a6004: Status 404 returned error can't find the container with id cc257cc0efbec4ccb1b6b93a1747a6d14c071b2b304234ef8c53fc822f9a6004 Sep 11 19:36:25.335853 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:25.335832 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:36:25.517641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:25.517606 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" event={"ID":"1e2c2640-d759-41bc-bc41-7f57b39cb976","Type":"ContainerStarted","Data":"cc257cc0efbec4ccb1b6b93a1747a6d14c071b2b304234ef8c53fc822f9a6004"} Sep 11 19:36:26.146166 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.146132 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r"] Sep 11 19:36:26.146356 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.146320 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" podUID="005e6b93-4a54-420f-bc2f-99c9525af30f" containerName="discovery" containerID="cri-o://7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0" gracePeriod=30 Sep 11 19:36:26.518990 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.518948 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:26.524194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.524167 2559 generic.go:356] "Generic (PLEG): container finished" podID="005e6b93-4a54-420f-bc2f-99c9525af30f" containerID="7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0" exitCode=0 Sep 11 19:36:26.524333 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.524178 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" Sep 11 19:36:26.524333 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.524195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" event={"ID":"005e6b93-4a54-420f-bc2f-99c9525af30f","Type":"ContainerDied","Data":"7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0"} Sep 11 19:36:26.524459 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.524350 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r" event={"ID":"005e6b93-4a54-420f-bc2f-99c9525af30f","Type":"ContainerDied","Data":"ad114b8601ecbda300afdfdc7032e807680c9dd34d6ed1d373ce9e55c1aed05b"} Sep 11 19:36:26.524459 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.524373 2559 scope.go:117] "RemoveContainer" containerID="7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0" Sep 11 19:36:26.592838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.592815 2559 scope.go:117] "RemoveContainer" containerID="7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0" Sep 11 19:36:26.593201 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:36:26.593169 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0\": container with ID starting with 7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0 not found: ID does not exist" containerID="7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0" Sep 11 19:36:26.593325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.593207 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0"} err="failed to get container status \"7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0\": rpc error: code = NotFound desc = could not find container \"7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0\": container with ID starting with 7f42943ec6d97f3137bd4815e9108e51586fe61a5a522143ef78bb032c9000a0 not found: ID does not exist" Sep 11 19:36:26.637720 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.637683 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-cacerts\") pod \"005e6b93-4a54-420f-bc2f-99c9525af30f\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " Sep 11 19:36:26.637902 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.637732 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005e6b93-4a54-420f-bc2f-99c9525af30f-local-certs\") pod \"005e6b93-4a54-420f-bc2f-99c9525af30f\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " Sep 11 19:36:26.637902 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.637767 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-kubeconfig\") pod \"005e6b93-4a54-420f-bc2f-99c9525af30f\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " Sep 11 19:36:26.637902 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.637798 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx572\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-kube-api-access-xx572\") pod \"005e6b93-4a54-420f-bc2f-99c9525af30f\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " Sep 11 19:36:26.637902 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.637868 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-token\") pod \"005e6b93-4a54-420f-bc2f-99c9525af30f\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " Sep 11 19:36:26.637902 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.637893 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-dns-cert\") pod \"005e6b93-4a54-420f-bc2f-99c9525af30f\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " Sep 11 19:36:26.638143 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.637928 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-ca-configmap\") pod \"005e6b93-4a54-420f-bc2f-99c9525af30f\" (UID: \"005e6b93-4a54-420f-bc2f-99c9525af30f\") " Sep 11 19:36:26.638610 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.638565 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "005e6b93-4a54-420f-bc2f-99c9525af30f" (UID: "005e6b93-4a54-420f-bc2f-99c9525af30f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:36:26.640654 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.640602 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-token" (OuterVolumeSpecName: "istio-token") pod "005e6b93-4a54-420f-bc2f-99c9525af30f" (UID: "005e6b93-4a54-420f-bc2f-99c9525af30f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:36:26.640989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.640964 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-cacerts" (OuterVolumeSpecName: "cacerts") pod "005e6b93-4a54-420f-bc2f-99c9525af30f" (UID: "005e6b93-4a54-420f-bc2f-99c9525af30f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:36:26.641143 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.640994 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "005e6b93-4a54-420f-bc2f-99c9525af30f" (UID: "005e6b93-4a54-420f-bc2f-99c9525af30f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:36:26.641551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.641503 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005e6b93-4a54-420f-bc2f-99c9525af30f-local-certs" (OuterVolumeSpecName: "local-certs") pod "005e6b93-4a54-420f-bc2f-99c9525af30f" (UID: "005e6b93-4a54-420f-bc2f-99c9525af30f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:36:26.641551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.641509 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "005e6b93-4a54-420f-bc2f-99c9525af30f" (UID: "005e6b93-4a54-420f-bc2f-99c9525af30f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:36:26.642331 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.642305 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-kube-api-access-xx572" (OuterVolumeSpecName: "kube-api-access-xx572") pod "005e6b93-4a54-420f-bc2f-99c9525af30f" (UID: "005e6b93-4a54-420f-bc2f-99c9525af30f"). InnerVolumeSpecName "kube-api-access-xx572". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:36:26.738989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.738953 2559 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-kubeconfig\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:36:26.738989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.738983 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx572\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-kube-api-access-xx572\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:36:26.738989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.738997 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-token\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:36:26.739219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.739011 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-dns-cert\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:36:26.739219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.739026 2559 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/005e6b93-4a54-420f-bc2f-99c9525af30f-istio-csr-ca-configmap\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:36:26.739219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.739039 2559 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/005e6b93-4a54-420f-bc2f-99c9525af30f-cacerts\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:36:26.739219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.739051 2559 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/005e6b93-4a54-420f-bc2f-99c9525af30f-local-certs\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:36:26.849671 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.849643 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r"] Sep 11 19:36:26.857854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:26.857827 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-56f5r"] Sep 11 19:36:27.222880 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:27.222811 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005e6b93-4a54-420f-bc2f-99c9525af30f" path="/var/lib/kubelet/pods/005e6b93-4a54-420f-bc2f-99c9525af30f/volumes" Sep 11 19:36:28.538614 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:28.538577 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" event={"ID":"1e2c2640-d759-41bc-bc41-7f57b39cb976","Type":"ContainerStarted","Data":"6f3ce508cab7e3af2ea688dda56778745945655609b8b2bba4cba0eddc8b49f3"} Sep 11 19:36:28.539100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:28.538689 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:36:28.561918 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:28.561868 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" podStartSLOduration=1.878882277 podStartE2EDuration="4.56185198s" totalImagesPullingTime="2.682969703s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:36:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:36:28.559333357 +0000 UTC m=+621.831840557" watchObservedRunningTime="2026-09-11 19:36:28.56185198 +0000 UTC m=+621.834359180" Sep 11 19:36:41.436059 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.436020 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9d68b4776-msw45"] Sep 11 19:36:41.436563 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.436328 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005e6b93-4a54-420f-bc2f-99c9525af30f" containerName="discovery" Sep 11 19:36:41.436563 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.436340 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="005e6b93-4a54-420f-bc2f-99c9525af30f" containerName="discovery" Sep 11 19:36:41.436563 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.436403 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="005e6b93-4a54-420f-bc2f-99c9525af30f" containerName="discovery" Sep 11 19:36:41.441448 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.441415 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.444084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.444062 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 19:36:41.444310 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.444298 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 19:36:41.445366 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.445343 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 19:36:41.445600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.445585 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 19:36:41.445691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.445619 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 19:36:41.445691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.445671 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vtmk8\"" Sep 11 19:36:41.445800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.445710 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 19:36:41.445885 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.445868 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 19:36:41.455049 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.455023 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d68b4776-msw45"] Sep 11 19:36:41.456229 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.456207 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 19:36:41.567416 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.567376 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-service-ca\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.567630 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.567429 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-oauth-config\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.567630 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.567523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-serving-cert\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.567630 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.567590 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-config\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.567630 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.567617 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-oauth-serving-cert\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.567797 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.567659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-trusted-ca-bundle\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.567797 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.567684 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5jr2\" (UniqueName: \"kubernetes.io/projected/b578b307-bf20-4d2a-b0e6-d7659b3645ae-kube-api-access-f5jr2\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.668854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.668810 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5jr2\" (UniqueName: \"kubernetes.io/projected/b578b307-bf20-4d2a-b0e6-d7659b3645ae-kube-api-access-f5jr2\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.668854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.668859 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-service-ca\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.669124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.668900 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-oauth-config\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.669124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.668936 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-serving-cert\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.669124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.668976 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-config\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.669124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.668999 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-oauth-serving-cert\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.669124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.669039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-trusted-ca-bundle\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.669712 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.669682 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-service-ca\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.669831 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.669723 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-oauth-serving-cert\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.669877 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.669771 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-config\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.670159 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.670137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b578b307-bf20-4d2a-b0e6-d7659b3645ae-trusted-ca-bundle\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.671616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.671590 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-oauth-config\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.671828 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.671805 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b578b307-bf20-4d2a-b0e6-d7659b3645ae-console-serving-cert\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.678510 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.678491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5jr2\" (UniqueName: \"kubernetes.io/projected/b578b307-bf20-4d2a-b0e6-d7659b3645ae-kube-api-access-f5jr2\") pod \"console-9d68b4776-msw45\" (UID: \"b578b307-bf20-4d2a-b0e6-d7659b3645ae\") " pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:41.752030 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:41.751997 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:42.025760 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:42.025735 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9d68b4776-msw45"] Sep 11 19:36:42.027739 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:36:42.027710 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb578b307_bf20_4d2a_b0e6_d7659b3645ae.slice/crio-4ce19f640187793c6f3a64ddd3c00affb7dd9f0bdbe39549ac4af0250eaf9b26 WatchSource:0}: Error finding container 4ce19f640187793c6f3a64ddd3c00affb7dd9f0bdbe39549ac4af0250eaf9b26: Status 404 returned error can't find the container with id 4ce19f640187793c6f3a64ddd3c00affb7dd9f0bdbe39549ac4af0250eaf9b26 Sep 11 19:36:42.604867 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:42.604831 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d68b4776-msw45" event={"ID":"b578b307-bf20-4d2a-b0e6-d7659b3645ae","Type":"ContainerStarted","Data":"4ce19f640187793c6f3a64ddd3c00affb7dd9f0bdbe39549ac4af0250eaf9b26"} Sep 11 19:36:45.624214 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:45.624182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9d68b4776-msw45" event={"ID":"b578b307-bf20-4d2a-b0e6-d7659b3645ae","Type":"ContainerStarted","Data":"c609b252863117a360925cea6b5d275b40959396ece0eafe9c4076fb692b11ca"} Sep 11 19:36:45.643626 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:45.643568 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9d68b4776-msw45" podStartSLOduration=1.175204838 podStartE2EDuration="4.643551791s" totalImagesPullingTime="3.468346953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:36:45.641977123 +0000 UTC m=+638.914484335" watchObservedRunningTime="2026-09-11 19:36:45.643551791 +0000 UTC m=+638.916058990" Sep 11 19:36:51.752643 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:51.752604 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:51.753019 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:51.752674 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:51.757470 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:51.757447 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:52.659846 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:52.659814 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9d68b4776-msw45" Sep 11 19:36:59.545008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:36:59.544974 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:37:19.317251 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.317211 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp"] Sep 11 19:37:19.322240 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.322223 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.325160 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.325130 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 19:37:19.325281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.325169 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 19:37:19.325281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.325200 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-p9dlz\"" Sep 11 19:37:19.330819 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.330797 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp"] Sep 11 19:37:19.423309 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.423275 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl"] Sep 11 19:37:19.425805 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.425781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.436507 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.436484 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl"] Sep 11 19:37:19.471487 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.471458 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzw5f\" (UniqueName: \"kubernetes.io/projected/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-kube-api-access-hzw5f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.471626 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.471512 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.471626 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.471561 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.515149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.515120 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65"] Sep 11 19:37:19.517717 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.517695 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.529011 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.528979 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65"] Sep 11 19:37:19.572935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.572854 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.572935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.572896 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.572935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.572918 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgwrn\" (UniqueName: \"kubernetes.io/projected/ee306a2d-bfa2-4589-b56a-21431163fe5c-kube-api-access-cgwrn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.573203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.573006 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.573203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.573055 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.573203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.573106 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzw5f\" (UniqueName: \"kubernetes.io/projected/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-kube-api-access-hzw5f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.573351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.573215 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.573351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.573282 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.582542 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.582514 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzw5f\" (UniqueName: \"kubernetes.io/projected/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-kube-api-access-hzw5f\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.617675 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.617643 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv"] Sep 11 19:37:19.620556 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.620536 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.630819 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.630798 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv"] Sep 11 19:37:19.632477 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.632459 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:19.674558 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.674523 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkrjr\" (UniqueName: \"kubernetes.io/projected/a93d095d-fd76-4d0d-a5ba-dfedd490083b-kube-api-access-fkrjr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.674741 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.674575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.674741 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.674593 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.674868 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.674743 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.674868 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.674778 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgwrn\" (UniqueName: \"kubernetes.io/projected/ee306a2d-bfa2-4589-b56a-21431163fe5c-kube-api-access-cgwrn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.674868 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.674801 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.675041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.675014 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.675158 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.675135 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.689923 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.689864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgwrn\" (UniqueName: \"kubernetes.io/projected/ee306a2d-bfa2-4589-b56a-21431163fe5c-kube-api-access-cgwrn\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.734079 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.734045 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:19.775744 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.775704 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.775956 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.775761 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.775956 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.775795 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzbr6\" (UniqueName: \"kubernetes.io/projected/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-kube-api-access-zzbr6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.775956 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.775916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.776063 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.775956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkrjr\" (UniqueName: \"kubernetes.io/projected/a93d095d-fd76-4d0d-a5ba-dfedd490083b-kube-api-access-fkrjr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.776063 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.775986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.776233 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.776211 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.776271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.776234 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.786877 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.786852 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkrjr\" (UniqueName: \"kubernetes.io/projected/a93d095d-fd76-4d0d-a5ba-dfedd490083b-kube-api-access-fkrjr\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.826968 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.826884 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:19.877086 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.877053 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.877244 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.877145 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.877244 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.877179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzbr6\" (UniqueName: \"kubernetes.io/projected/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-kube-api-access-zzbr6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.877538 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.877516 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.877793 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.877764 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.888389 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.888365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzbr6\" (UniqueName: \"kubernetes.io/projected/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-kube-api-access-zzbr6\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:19.920103 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.920074 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp"] Sep 11 19:37:19.922728 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:37:19.922703 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac2633a_c76e_48ee_9edf_fb2d500a4aa4.slice/crio-5d8b874f0c383762e7042949aeaf8aea4f02e4101405addac8ca6e4ab64d58b3 WatchSource:0}: Error finding container 5d8b874f0c383762e7042949aeaf8aea4f02e4101405addac8ca6e4ab64d58b3: Status 404 returned error can't find the container with id 5d8b874f0c383762e7042949aeaf8aea4f02e4101405addac8ca6e4ab64d58b3 Sep 11 19:37:19.930514 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:19.930496 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:20.017529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.017494 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl"] Sep 11 19:37:20.018891 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:37:20.018861 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee306a2d_bfa2_4589_b56a_21431163fe5c.slice/crio-8caff985e9093756e8908d4a18919fd53ee93eb572a2261e703462f72d7cdcbe WatchSource:0}: Error finding container 8caff985e9093756e8908d4a18919fd53ee93eb572a2261e703462f72d7cdcbe: Status 404 returned error can't find the container with id 8caff985e9093756e8908d4a18919fd53ee93eb572a2261e703462f72d7cdcbe Sep 11 19:37:20.101778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.101755 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65"] Sep 11 19:37:20.119406 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:37:20.119376 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93d095d_fd76_4d0d_a5ba_dfedd490083b.slice/crio-66d86daf4b3e5a84cb6799f9deea1322bb446ba8e6beb810774d07931b85284c WatchSource:0}: Error finding container 66d86daf4b3e5a84cb6799f9deea1322bb446ba8e6beb810774d07931b85284c: Status 404 returned error can't find the container with id 66d86daf4b3e5a84cb6799f9deea1322bb446ba8e6beb810774d07931b85284c Sep 11 19:37:20.212149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.211989 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv"] Sep 11 19:37:20.292298 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:37:20.292251 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c2ae8b_4745_4a4c_9586_a5ea71cf5eb1.slice/crio-7512a1e8ecb98eb77118ed0fe99a3706ffb7038f27feeaa6e5f7598d7be7d7a2 WatchSource:0}: Error finding container 7512a1e8ecb98eb77118ed0fe99a3706ffb7038f27feeaa6e5f7598d7be7d7a2: Status 404 returned error can't find the container with id 7512a1e8ecb98eb77118ed0fe99a3706ffb7038f27feeaa6e5f7598d7be7d7a2 Sep 11 19:37:20.790021 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.789984 2559 generic.go:356] "Generic (PLEG): container finished" podID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerID="85345dda5bb2cccbd680f7e38ce01c893cfe05c12f59f60e6c8548666a3a4a64" exitCode=0 Sep 11 19:37:20.790422 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.790066 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" event={"ID":"a93d095d-fd76-4d0d-a5ba-dfedd490083b","Type":"ContainerDied","Data":"85345dda5bb2cccbd680f7e38ce01c893cfe05c12f59f60e6c8548666a3a4a64"} Sep 11 19:37:20.790422 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.790117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" event={"ID":"a93d095d-fd76-4d0d-a5ba-dfedd490083b","Type":"ContainerStarted","Data":"66d86daf4b3e5a84cb6799f9deea1322bb446ba8e6beb810774d07931b85284c"} Sep 11 19:37:20.791719 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.791698 2559 generic.go:356] "Generic (PLEG): container finished" podID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerID="ec9045e030f2e14881ac8d6eeb9b97e4ffff5039a6527436b32d8d950cfce894" exitCode=0 Sep 11 19:37:20.791809 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.791784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" event={"ID":"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1","Type":"ContainerDied","Data":"ec9045e030f2e14881ac8d6eeb9b97e4ffff5039a6527436b32d8d950cfce894"} Sep 11 19:37:20.791867 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.791835 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" event={"ID":"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1","Type":"ContainerStarted","Data":"7512a1e8ecb98eb77118ed0fe99a3706ffb7038f27feeaa6e5f7598d7be7d7a2"} Sep 11 19:37:20.793433 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.793412 2559 generic.go:356] "Generic (PLEG): container finished" podID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerID="92d1220fa4f5ad054a52f39b4cb044137b424e2e61b1adedb85a83dc9ddb891e" exitCode=0 Sep 11 19:37:20.793516 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.793473 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" event={"ID":"ee306a2d-bfa2-4589-b56a-21431163fe5c","Type":"ContainerDied","Data":"92d1220fa4f5ad054a52f39b4cb044137b424e2e61b1adedb85a83dc9ddb891e"} Sep 11 19:37:20.793516 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.793511 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" event={"ID":"ee306a2d-bfa2-4589-b56a-21431163fe5c","Type":"ContainerStarted","Data":"8caff985e9093756e8908d4a18919fd53ee93eb572a2261e703462f72d7cdcbe"} Sep 11 19:37:20.795152 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.795132 2559 generic.go:356] "Generic (PLEG): container finished" podID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerID="0967b6f10b0f4d80fee955e5193b9650d3d646eac8705477f60f87d552737ff0" exitCode=0 Sep 11 19:37:20.795250 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.795192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" event={"ID":"dac2633a-c76e-48ee-9edf-fb2d500a4aa4","Type":"ContainerDied","Data":"0967b6f10b0f4d80fee955e5193b9650d3d646eac8705477f60f87d552737ff0"} Sep 11 19:37:20.795250 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:20.795223 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" event={"ID":"dac2633a-c76e-48ee-9edf-fb2d500a4aa4","Type":"ContainerStarted","Data":"5d8b874f0c383762e7042949aeaf8aea4f02e4101405addac8ca6e4ab64d58b3"} Sep 11 19:37:22.806550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:22.806514 2559 generic.go:356] "Generic (PLEG): container finished" podID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerID="d7f379df22d8a771c6c287350b18666abe372d5f5b2be293e5d60f2841af0a4d" exitCode=0 Sep 11 19:37:22.806967 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:22.806593 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" event={"ID":"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1","Type":"ContainerDied","Data":"d7f379df22d8a771c6c287350b18666abe372d5f5b2be293e5d60f2841af0a4d"} Sep 11 19:37:22.808550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:22.808524 2559 generic.go:356] "Generic (PLEG): container finished" podID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerID="28d1fd675cfd380613eb397fa7ee72919159b605eca8a98ec9caa2e904387070" exitCode=0 Sep 11 19:37:22.808668 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:22.808609 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" event={"ID":"ee306a2d-bfa2-4589-b56a-21431163fe5c","Type":"ContainerDied","Data":"28d1fd675cfd380613eb397fa7ee72919159b605eca8a98ec9caa2e904387070"} Sep 11 19:37:22.810862 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:22.810841 2559 generic.go:356] "Generic (PLEG): container finished" podID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerID="76d504a67a25c88673fc30336bd4008096ceb41442efeb0a611e256a129ea314" exitCode=0 Sep 11 19:37:22.810958 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:22.810928 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" event={"ID":"dac2633a-c76e-48ee-9edf-fb2d500a4aa4","Type":"ContainerDied","Data":"76d504a67a25c88673fc30336bd4008096ceb41442efeb0a611e256a129ea314"} Sep 11 19:37:22.813032 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:22.813013 2559 generic.go:356] "Generic (PLEG): container finished" podID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerID="1ba1946346202501a513fe1dad57e87915f856c38d8e3a1e0d21e215a4fe9298" exitCode=0 Sep 11 19:37:22.813117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:22.813045 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" event={"ID":"a93d095d-fd76-4d0d-a5ba-dfedd490083b","Type":"ContainerDied","Data":"1ba1946346202501a513fe1dad57e87915f856c38d8e3a1e0d21e215a4fe9298"} Sep 11 19:37:23.820588 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:23.820555 2559 generic.go:356] "Generic (PLEG): container finished" podID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerID="68b2d2f259715fb546be13aa903b0dfed665d9fa41bcc4ccae2b7c53e8d44023" exitCode=0 Sep 11 19:37:23.821010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:23.820645 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" event={"ID":"ee306a2d-bfa2-4589-b56a-21431163fe5c","Type":"ContainerDied","Data":"68b2d2f259715fb546be13aa903b0dfed665d9fa41bcc4ccae2b7c53e8d44023"} Sep 11 19:37:23.822753 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:23.822731 2559 generic.go:356] "Generic (PLEG): container finished" podID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerID="123e6d6db5721a13e2946ee25280727d9339a05af79ed025e2d3cfa300190cea" exitCode=0 Sep 11 19:37:23.822846 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:23.822815 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" event={"ID":"dac2633a-c76e-48ee-9edf-fb2d500a4aa4","Type":"ContainerDied","Data":"123e6d6db5721a13e2946ee25280727d9339a05af79ed025e2d3cfa300190cea"} Sep 11 19:37:23.824983 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:23.824964 2559 generic.go:356] "Generic (PLEG): container finished" podID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerID="074ce47d8cd508eaab148becd555053f379940467e5d34139b4311db97e09790" exitCode=0 Sep 11 19:37:23.825066 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:23.825040 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" event={"ID":"a93d095d-fd76-4d0d-a5ba-dfedd490083b","Type":"ContainerDied","Data":"074ce47d8cd508eaab148becd555053f379940467e5d34139b4311db97e09790"} Sep 11 19:37:23.827031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:23.827012 2559 generic.go:356] "Generic (PLEG): container finished" podID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerID="b52a4f4baaa889593c758d8b1e9bb9b8c98449cc5a0bab44f83592a3afb47bc1" exitCode=0 Sep 11 19:37:23.827155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:23.827048 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" event={"ID":"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1","Type":"ContainerDied","Data":"b52a4f4baaa889593c758d8b1e9bb9b8c98449cc5a0bab44f83592a3afb47bc1"} Sep 11 19:37:24.991113 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:24.991084 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:25.040739 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.040714 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:25.043590 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.043571 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:25.046743 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.046725 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:25.132608 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132524 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgwrn\" (UniqueName: \"kubernetes.io/projected/ee306a2d-bfa2-4589-b56a-21431163fe5c-kube-api-access-cgwrn\") pod \"ee306a2d-bfa2-4589-b56a-21431163fe5c\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " Sep 11 19:37:25.132608 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132579 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzw5f\" (UniqueName: \"kubernetes.io/projected/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-kube-api-access-hzw5f\") pod \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " Sep 11 19:37:25.132845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132625 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-util\") pod \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " Sep 11 19:37:25.132845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132670 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-util\") pod \"ee306a2d-bfa2-4589-b56a-21431163fe5c\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " Sep 11 19:37:25.132845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132712 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-util\") pod \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " Sep 11 19:37:25.132845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132743 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-util\") pod \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " Sep 11 19:37:25.132845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132773 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkrjr\" (UniqueName: \"kubernetes.io/projected/a93d095d-fd76-4d0d-a5ba-dfedd490083b-kube-api-access-fkrjr\") pod \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " Sep 11 19:37:25.132845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132814 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-bundle\") pod \"ee306a2d-bfa2-4589-b56a-21431163fe5c\" (UID: \"ee306a2d-bfa2-4589-b56a-21431163fe5c\") " Sep 11 19:37:25.132845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132837 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zzbr6\" (UniqueName: \"kubernetes.io/projected/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-kube-api-access-zzbr6\") pod \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " Sep 11 19:37:25.133161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132870 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-bundle\") pod \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\" (UID: \"dac2633a-c76e-48ee-9edf-fb2d500a4aa4\") " Sep 11 19:37:25.133161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132899 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-bundle\") pod \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\" (UID: \"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1\") " Sep 11 19:37:25.133161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.132922 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-bundle\") pod \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\" (UID: \"a93d095d-fd76-4d0d-a5ba-dfedd490083b\") " Sep 11 19:37:25.133968 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.133783 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-bundle" (OuterVolumeSpecName: "bundle") pod "a93d095d-fd76-4d0d-a5ba-dfedd490083b" (UID: "a93d095d-fd76-4d0d-a5ba-dfedd490083b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:37:25.134579 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.134525 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-bundle" (OuterVolumeSpecName: "bundle") pod "dac2633a-c76e-48ee-9edf-fb2d500a4aa4" (UID: "dac2633a-c76e-48ee-9edf-fb2d500a4aa4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:37:25.134698 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.134590 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-bundle" (OuterVolumeSpecName: "bundle") pod "e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" (UID: "e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:37:25.135506 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.135463 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-bundle" (OuterVolumeSpecName: "bundle") pod "ee306a2d-bfa2-4589-b56a-21431163fe5c" (UID: "ee306a2d-bfa2-4589-b56a-21431163fe5c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:37:25.135657 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.135626 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-kube-api-access-hzw5f" (OuterVolumeSpecName: "kube-api-access-hzw5f") pod "dac2633a-c76e-48ee-9edf-fb2d500a4aa4" (UID: "dac2633a-c76e-48ee-9edf-fb2d500a4aa4"). InnerVolumeSpecName "kube-api-access-hzw5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:37:25.135883 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.135856 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee306a2d-bfa2-4589-b56a-21431163fe5c-kube-api-access-cgwrn" (OuterVolumeSpecName: "kube-api-access-cgwrn") pod "ee306a2d-bfa2-4589-b56a-21431163fe5c" (UID: "ee306a2d-bfa2-4589-b56a-21431163fe5c"). InnerVolumeSpecName "kube-api-access-cgwrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:37:25.136151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.136136 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93d095d-fd76-4d0d-a5ba-dfedd490083b-kube-api-access-fkrjr" (OuterVolumeSpecName: "kube-api-access-fkrjr") pod "a93d095d-fd76-4d0d-a5ba-dfedd490083b" (UID: "a93d095d-fd76-4d0d-a5ba-dfedd490083b"). InnerVolumeSpecName "kube-api-access-fkrjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:37:25.136555 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.136536 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-kube-api-access-zzbr6" (OuterVolumeSpecName: "kube-api-access-zzbr6") pod "e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" (UID: "e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1"). InnerVolumeSpecName "kube-api-access-zzbr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:37:25.139351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.139313 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-util" (OuterVolumeSpecName: "util") pod "a93d095d-fd76-4d0d-a5ba-dfedd490083b" (UID: "a93d095d-fd76-4d0d-a5ba-dfedd490083b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:37:25.141741 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.141712 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-util" (OuterVolumeSpecName: "util") pod "ee306a2d-bfa2-4589-b56a-21431163fe5c" (UID: "ee306a2d-bfa2-4589-b56a-21431163fe5c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:37:25.141814 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.141792 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-util" (OuterVolumeSpecName: "util") pod "e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" (UID: "e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:37:25.207090 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.207057 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-util" (OuterVolumeSpecName: "util") pod "dac2633a-c76e-48ee-9edf-fb2d500a4aa4" (UID: "dac2633a-c76e-48ee-9edf-fb2d500a4aa4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234382 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234402 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzbr6\" (UniqueName: \"kubernetes.io/projected/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-kube-api-access-zzbr6\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234412 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234421 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234429 2559 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-bundle\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234453 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgwrn\" (UniqueName: \"kubernetes.io/projected/ee306a2d-bfa2-4589-b56a-21431163fe5c-kube-api-access-cgwrn\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234463 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzw5f\" (UniqueName: \"kubernetes.io/projected/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-kube-api-access-hzw5f\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234472 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dac2633a-c76e-48ee-9edf-fb2d500a4aa4-util\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234479 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee306a2d-bfa2-4589-b56a-21431163fe5c-util\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234486 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1-util\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234492 2559 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a93d095d-fd76-4d0d-a5ba-dfedd490083b-util\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.234600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.234499 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkrjr\" (UniqueName: \"kubernetes.io/projected/a93d095d-fd76-4d0d-a5ba-dfedd490083b-kube-api-access-fkrjr\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:37:25.839307 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.839281 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" Sep 11 19:37:25.839307 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.839290 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d74zs65" event={"ID":"a93d095d-fd76-4d0d-a5ba-dfedd490083b","Type":"ContainerDied","Data":"66d86daf4b3e5a84cb6799f9deea1322bb446ba8e6beb810774d07931b85284c"} Sep 11 19:37:25.839555 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.839337 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66d86daf4b3e5a84cb6799f9deea1322bb446ba8e6beb810774d07931b85284c" Sep 11 19:37:25.841500 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.841471 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" event={"ID":"e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1","Type":"ContainerDied","Data":"7512a1e8ecb98eb77118ed0fe99a3706ffb7038f27feeaa6e5f7598d7be7d7a2"} Sep 11 19:37:25.841625 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.841516 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7512a1e8ecb98eb77118ed0fe99a3706ffb7038f27feeaa6e5f7598d7be7d7a2" Sep 11 19:37:25.841625 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.841481 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c65z8kv" Sep 11 19:37:25.843672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.843653 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" Sep 11 19:37:25.843785 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.843650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847aqx8wl" event={"ID":"ee306a2d-bfa2-4589-b56a-21431163fe5c","Type":"ContainerDied","Data":"8caff985e9093756e8908d4a18919fd53ee93eb572a2261e703462f72d7cdcbe"} Sep 11 19:37:25.843785 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.843764 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8caff985e9093756e8908d4a18919fd53ee93eb572a2261e703462f72d7cdcbe" Sep 11 19:37:25.845810 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.845782 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" Sep 11 19:37:25.845905 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.845779 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cw4mdp" event={"ID":"dac2633a-c76e-48ee-9edf-fb2d500a4aa4","Type":"ContainerDied","Data":"5d8b874f0c383762e7042949aeaf8aea4f02e4101405addac8ca6e4ab64d58b3"} Sep 11 19:37:25.845972 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:25.845907 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d8b874f0c383762e7042949aeaf8aea4f02e4101405addac8ca6e4ab64d58b3" Sep 11 19:37:32.748507 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748470 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fjhz5"] Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748772 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerName="pull" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748783 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerName="pull" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748790 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerName="pull" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748795 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerName="pull" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748805 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerName="util" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748810 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerName="util" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748816 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerName="pull" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748822 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerName="pull" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748830 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerName="util" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748835 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerName="util" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748841 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerName="extract" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748846 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerName="extract" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748853 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerName="util" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748857 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerName="util" Sep 11 19:37:32.748857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748863 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748870 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748876 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748881 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748887 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerName="pull" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748893 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerName="pull" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748899 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748905 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748913 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerName="util" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748918 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerName="util" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748969 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a93d095d-fd76-4d0d-a5ba-dfedd490083b" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748976 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2c2ae8b-4745-4a4c-9586-a5ea71cf5eb1" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748983 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dac2633a-c76e-48ee-9edf-fb2d500a4aa4" containerName="extract" Sep 11 19:37:32.749265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.748988 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee306a2d-bfa2-4589-b56a-21431163fe5c" containerName="extract" Sep 11 19:37:32.751854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.751838 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" Sep 11 19:37:32.754537 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.754512 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 19:37:32.754672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.754564 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-nflk7\"" Sep 11 19:37:32.754672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.754580 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 19:37:32.762337 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.762315 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fjhz5"] Sep 11 19:37:32.897825 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.897788 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx5lq\" (UniqueName: \"kubernetes.io/projected/274c0e01-91ab-430b-8abc-ce22883d0705-kube-api-access-zx5lq\") pod \"authorino-operator-76b478bd78-fjhz5\" (UID: \"274c0e01-91ab-430b-8abc-ce22883d0705\") " pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" Sep 11 19:37:32.999213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:32.999126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx5lq\" (UniqueName: \"kubernetes.io/projected/274c0e01-91ab-430b-8abc-ce22883d0705-kube-api-access-zx5lq\") pod \"authorino-operator-76b478bd78-fjhz5\" (UID: \"274c0e01-91ab-430b-8abc-ce22883d0705\") " pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" Sep 11 19:37:33.009942 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:33.009916 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx5lq\" (UniqueName: \"kubernetes.io/projected/274c0e01-91ab-430b-8abc-ce22883d0705-kube-api-access-zx5lq\") pod \"authorino-operator-76b478bd78-fjhz5\" (UID: \"274c0e01-91ab-430b-8abc-ce22883d0705\") " pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" Sep 11 19:37:33.062526 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:33.062491 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" Sep 11 19:37:33.339809 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:33.339782 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-fjhz5"] Sep 11 19:37:33.341570 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:37:33.341543 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod274c0e01_91ab_430b_8abc_ce22883d0705.slice/crio-ea8ed44f9311558a70dc84307b76ca83272990ff35c4575c324bad0edb3af901 WatchSource:0}: Error finding container ea8ed44f9311558a70dc84307b76ca83272990ff35c4575c324bad0edb3af901: Status 404 returned error can't find the container with id ea8ed44f9311558a70dc84307b76ca83272990ff35c4575c324bad0edb3af901 Sep 11 19:37:33.891951 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:33.891912 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" event={"ID":"274c0e01-91ab-430b-8abc-ce22883d0705","Type":"ContainerStarted","Data":"ea8ed44f9311558a70dc84307b76ca83272990ff35c4575c324bad0edb3af901"} Sep 11 19:37:37.073641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.073602 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r"] Sep 11 19:37:37.077566 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.077543 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:37.081680 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.081661 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5xldn\"" Sep 11 19:37:37.098394 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.098362 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r"] Sep 11 19:37:37.135151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.135116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca36913a-a511-4561-8787-e824bf6eb6e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" (UID: \"ca36913a-a511-4561-8787-e824bf6eb6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:37.135334 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.135156 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv762\" (UniqueName: \"kubernetes.io/projected/ca36913a-a511-4561-8787-e824bf6eb6e7-kube-api-access-jv762\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" (UID: \"ca36913a-a511-4561-8787-e824bf6eb6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:37.236142 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.236109 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca36913a-a511-4561-8787-e824bf6eb6e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" (UID: \"ca36913a-a511-4561-8787-e824bf6eb6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:37.236142 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.236144 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv762\" (UniqueName: \"kubernetes.io/projected/ca36913a-a511-4561-8787-e824bf6eb6e7-kube-api-access-jv762\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" (UID: \"ca36913a-a511-4561-8787-e824bf6eb6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:37.236570 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.236544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca36913a-a511-4561-8787-e824bf6eb6e7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" (UID: \"ca36913a-a511-4561-8787-e824bf6eb6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:37.246097 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.246074 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv762\" (UniqueName: \"kubernetes.io/projected/ca36913a-a511-4561-8787-e824bf6eb6e7-kube-api-access-jv762\") pod \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" (UID: \"ca36913a-a511-4561-8787-e824bf6eb6e7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:37.389459 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.389410 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:37.675897 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.675869 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r"] Sep 11 19:37:37.677125 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:37:37.677093 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca36913a_a511_4561_8787_e824bf6eb6e7.slice/crio-c0e52c871b9ce71681dc8d827cf411a2ce28385d31ea104bd951f7a9e4764b79 WatchSource:0}: Error finding container c0e52c871b9ce71681dc8d827cf411a2ce28385d31ea104bd951f7a9e4764b79: Status 404 returned error can't find the container with id c0e52c871b9ce71681dc8d827cf411a2ce28385d31ea104bd951f7a9e4764b79 Sep 11 19:37:37.917733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.917692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" event={"ID":"274c0e01-91ab-430b-8abc-ce22883d0705","Type":"ContainerStarted","Data":"76b2888499f8e5b61cb5d56b40995b5581c3772a132053b6784a948b56540b66"} Sep 11 19:37:37.917925 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.917757 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" Sep 11 19:37:37.919143 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.919113 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" event={"ID":"ca36913a-a511-4561-8787-e824bf6eb6e7","Type":"ContainerStarted","Data":"c0e52c871b9ce71681dc8d827cf411a2ce28385d31ea104bd951f7a9e4764b79"} Sep 11 19:37:37.949931 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:37.949826 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" podStartSLOduration=1.9836381429999999 podStartE2EDuration="5.949809201s" totalImagesPullingTime="3.966171058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:37:37.949533391 +0000 UTC m=+691.222040592" watchObservedRunningTime="2026-09-11 19:37:37.949809201 +0000 UTC m=+691.222316398" Sep 11 19:37:39.071990 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.071953 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr"] Sep 11 19:37:39.076407 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.076391 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:37:39.082826 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.082793 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-nhf87\"" Sep 11 19:37:39.087475 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.087432 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr"] Sep 11 19:37:39.152560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.152529 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82kd6\" (UniqueName: \"kubernetes.io/projected/460c8ca1-752e-4cc4-9206-9b705adbdcde-kube-api-access-82kd6\") pod \"limitador-operator-controller-manager-74495c69f8-t9rsr\" (UID: \"460c8ca1-752e-4cc4-9206-9b705adbdcde\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:37:39.253380 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.253352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82kd6\" (UniqueName: \"kubernetes.io/projected/460c8ca1-752e-4cc4-9206-9b705adbdcde-kube-api-access-82kd6\") pod \"limitador-operator-controller-manager-74495c69f8-t9rsr\" (UID: \"460c8ca1-752e-4cc4-9206-9b705adbdcde\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:37:39.295071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.295031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82kd6\" (UniqueName: \"kubernetes.io/projected/460c8ca1-752e-4cc4-9206-9b705adbdcde-kube-api-access-82kd6\") pod \"limitador-operator-controller-manager-74495c69f8-t9rsr\" (UID: \"460c8ca1-752e-4cc4-9206-9b705adbdcde\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:37:39.387226 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.387138 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:37:39.664392 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.664361 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr"] Sep 11 19:37:39.666107 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:37:39.666082 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod460c8ca1_752e_4cc4_9206_9b705adbdcde.slice/crio-0e407e70d46214a5f18255348fc904f18b83b2d9fc779440a254bc16bdbe10ca WatchSource:0}: Error finding container 0e407e70d46214a5f18255348fc904f18b83b2d9fc779440a254bc16bdbe10ca: Status 404 returned error can't find the container with id 0e407e70d46214a5f18255348fc904f18b83b2d9fc779440a254bc16bdbe10ca Sep 11 19:37:39.930225 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:39.930140 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" event={"ID":"460c8ca1-752e-4cc4-9206-9b705adbdcde","Type":"ContainerStarted","Data":"0e407e70d46214a5f18255348fc904f18b83b2d9fc779440a254bc16bdbe10ca"} Sep 11 19:37:41.941882 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:41.941784 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" event={"ID":"460c8ca1-752e-4cc4-9206-9b705adbdcde","Type":"ContainerStarted","Data":"23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af"} Sep 11 19:37:41.942227 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:41.941922 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:37:41.960201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:41.960152 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" podStartSLOduration=1.027613858 podStartE2EDuration="2.960137217s" totalImagesPullingTime="1.932523359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:37:41.958676824 +0000 UTC m=+695.231184024" watchObservedRunningTime="2026-09-11 19:37:41.960137217 +0000 UTC m=+695.232644418" Sep 11 19:37:48.926303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:48.926264 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-fjhz5" Sep 11 19:37:50.993152 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:50.993108 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" event={"ID":"ca36913a-a511-4561-8787-e824bf6eb6e7","Type":"ContainerStarted","Data":"46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa"} Sep 11 19:37:50.993691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:50.993252 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:37:51.021492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:51.021419 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" podStartSLOduration=1.252431204 podStartE2EDuration="14.021403035s" totalImagesPullingTime="12.768971831s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:37:51.018656746 +0000 UTC m=+704.291163946" watchObservedRunningTime="2026-09-11 19:37:51.021403035 +0000 UTC m=+704.293910235" Sep 11 19:37:52.941332 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:52.941293 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9"] Sep 11 19:37:52.947770 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:52.947743 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:52.949804 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:52.949776 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:37:52.950747 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:52.950725 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-klrmj\"" Sep 11 19:37:52.950865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:52.950760 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 11 19:37:52.950865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:52.950766 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 11 19:37:52.952076 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:52.952052 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9"] Sep 11 19:37:53.071273 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.071240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec4b84da-6719-4343-9536-b324581bac1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.071525 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.071293 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec4b84da-6719-4343-9536-b324581bac1c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.071525 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.071366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj4sd\" (UniqueName: \"kubernetes.io/projected/ec4b84da-6719-4343-9536-b324581bac1c-kube-api-access-cj4sd\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.172605 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.172564 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec4b84da-6719-4343-9536-b324581bac1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.172800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.172612 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec4b84da-6719-4343-9536-b324581bac1c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.172800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.172669 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj4sd\" (UniqueName: \"kubernetes.io/projected/ec4b84da-6719-4343-9536-b324581bac1c-kube-api-access-cj4sd\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.172800 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:37:53.172729 2559 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 11 19:37:53.172800 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:37:53.172802 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec4b84da-6719-4343-9536-b324581bac1c-plugin-serving-cert podName:ec4b84da-6719-4343-9536-b324581bac1c nodeName:}" failed. No retries permitted until 2026-09-11 19:37:53.672781112 +0000 UTC m=+706.945288301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/ec4b84da-6719-4343-9536-b324581bac1c-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-qrst9" (UID: "ec4b84da-6719-4343-9536-b324581bac1c") : secret "plugin-serving-cert" not found Sep 11 19:37:53.173380 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.173362 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ec4b84da-6719-4343-9536-b324581bac1c-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.183481 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.183431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj4sd\" (UniqueName: \"kubernetes.io/projected/ec4b84da-6719-4343-9536-b324581bac1c-kube-api-access-cj4sd\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.679001 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.678960 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec4b84da-6719-4343-9536-b324581bac1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.681550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.681530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ec4b84da-6719-4343-9536-b324581bac1c-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-qrst9\" (UID: \"ec4b84da-6719-4343-9536-b324581bac1c\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:53.858213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:53.858171 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" Sep 11 19:37:54.132723 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:54.132698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9"] Sep 11 19:37:54.133792 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:37:54.133766 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec4b84da_6719_4343_9536_b324581bac1c.slice/crio-d831e14f617eca7a57f720925a8eb1d525081a5e0d8d84c9e352185697907be5 WatchSource:0}: Error finding container d831e14f617eca7a57f720925a8eb1d525081a5e0d8d84c9e352185697907be5: Status 404 returned error can't find the container with id d831e14f617eca7a57f720925a8eb1d525081a5e0d8d84c9e352185697907be5 Sep 11 19:37:55.017234 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:55.017173 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" event={"ID":"ec4b84da-6719-4343-9536-b324581bac1c","Type":"ContainerStarted","Data":"d831e14f617eca7a57f720925a8eb1d525081a5e0d8d84c9e352185697907be5"} Sep 11 19:37:59.043632 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:59.043597 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" event={"ID":"ec4b84da-6719-4343-9536-b324581bac1c","Type":"ContainerStarted","Data":"2faaf930d0f9dfd07b84b817aa676059ffd1785c393cf2ea6ae68879b96bab54"} Sep 11 19:37:59.061529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:37:59.061470 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-qrst9" podStartSLOduration=2.462130874 podStartE2EDuration="7.06145335s" totalImagesPullingTime="4.599322476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:37:59.059770047 +0000 UTC m=+712.332277239" watchObservedRunningTime="2026-09-11 19:37:59.06145335 +0000 UTC m=+712.333960594" Sep 11 19:38:02.000825 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:02.000792 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:03.651859 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.651810 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4"] Sep 11 19:38:03.654710 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.654693 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:03.671840 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.671814 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4"] Sep 11 19:38:03.775116 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.775076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-289cf\" (UniqueName: \"kubernetes.io/projected/130befcf-2f3f-4de4-88c5-6390d88f2b01-kube-api-access-289cf\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" (UID: \"130befcf-2f3f-4de4-88c5-6390d88f2b01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:03.775301 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.775139 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/130befcf-2f3f-4de4-88c5-6390d88f2b01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" (UID: \"130befcf-2f3f-4de4-88c5-6390d88f2b01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:03.876254 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.876214 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/130befcf-2f3f-4de4-88c5-6390d88f2b01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" (UID: \"130befcf-2f3f-4de4-88c5-6390d88f2b01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:03.876497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.876298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-289cf\" (UniqueName: \"kubernetes.io/projected/130befcf-2f3f-4de4-88c5-6390d88f2b01-kube-api-access-289cf\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" (UID: \"130befcf-2f3f-4de4-88c5-6390d88f2b01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:03.876735 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.876710 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/130befcf-2f3f-4de4-88c5-6390d88f2b01-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" (UID: \"130befcf-2f3f-4de4-88c5-6390d88f2b01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:03.885573 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.885554 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-289cf\" (UniqueName: \"kubernetes.io/projected/130befcf-2f3f-4de4-88c5-6390d88f2b01-kube-api-access-289cf\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" (UID: \"130befcf-2f3f-4de4-88c5-6390d88f2b01\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:03.965258 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:03.965173 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:04.108243 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.108202 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r"] Sep 11 19:38:04.108526 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.108493 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" containerName="manager" containerID="cri-o://46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa" gracePeriod=2 Sep 11 19:38:04.112162 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.112118 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:04.113171 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.113148 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r"] Sep 11 19:38:04.130832 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.130771 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4"] Sep 11 19:38:04.133299 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.133271 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4"] Sep 11 19:38:04.152928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.152899 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr"] Sep 11 19:38:04.153133 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.153093 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" containerName="manager" containerID="cri-o://23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af" gracePeriod=2 Sep 11 19:38:04.178076 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.178052 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr"] Sep 11 19:38:04.192021 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.191990 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh"] Sep 11 19:38:04.192370 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.192354 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" containerName="manager" Sep 11 19:38:04.192463 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.192372 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" containerName="manager" Sep 11 19:38:04.192463 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.192396 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" containerName="manager" Sep 11 19:38:04.192463 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.192404 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" containerName="manager" Sep 11 19:38:04.192616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.192517 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" containerName="manager" Sep 11 19:38:04.192616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.192532 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" containerName="manager" Sep 11 19:38:04.195269 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.195251 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:04.200037 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.199996 2559 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"460c8ca1-752e-4cc4-9206-9b705adbdcde\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-t9rsr\": pods \"limitador-operator-controller-manager-74495c69f8-t9rsr\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:04.204401 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.204371 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:04.207321 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.207293 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:04.209609 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.209587 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t9rsr\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:04.222108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.222046 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59"] Sep 11 19:38:04.225465 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.225428 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" Sep 11 19:38:04.231032 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.230635 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh"] Sep 11 19:38:04.243565 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.243542 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59"] Sep 11 19:38:04.259329 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.259289 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:04.280553 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.280505 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51f06832-aba7-4956-8da6-db3b8b20284c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ml6bh\" (UID: \"51f06832-aba7-4956-8da6-db3b8b20284c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:04.280705 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.280584 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scsjg\" (UniqueName: \"kubernetes.io/projected/51f06832-aba7-4956-8da6-db3b8b20284c-kube-api-access-scsjg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ml6bh\" (UID: \"51f06832-aba7-4956-8da6-db3b8b20284c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:04.303677 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.303630 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t9rsr\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:04.381391 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.381314 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51f06832-aba7-4956-8da6-db3b8b20284c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ml6bh\" (UID: \"51f06832-aba7-4956-8da6-db3b8b20284c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:04.381497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.381397 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scsjg\" (UniqueName: \"kubernetes.io/projected/51f06832-aba7-4956-8da6-db3b8b20284c-kube-api-access-scsjg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ml6bh\" (UID: \"51f06832-aba7-4956-8da6-db3b8b20284c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:04.381497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.381427 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f2b5\" (UniqueName: \"kubernetes.io/projected/2818e853-46df-4e26-9c31-0f77fdf9c5c8-kube-api-access-6f2b5\") pod \"limitador-operator-controller-manager-74495c69f8-prh59\" (UID: \"2818e853-46df-4e26-9c31-0f77fdf9c5c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" Sep 11 19:38:04.381646 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.381596 2559 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 11 19:38:04.381646 ip-10-0-137-97 kubenswrapper[2559]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-8zkj4_kuadrant-system_130befcf-2f3f-4de4-88c5-6390d88f2b01_0(3c607e23e4f51d317f82b2652ba3318dbb8385ee0cc12e1a7618d1f26256aeb9): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-8zkj4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3c607e23e4f51d317f82b2652ba3318dbb8385ee0cc12e1a7618d1f26256aeb9" Netns:"/var/run/netns/5d226ea9-8b2a-4980-af3b-6bbc8ab779a5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-8zkj4;K8S_POD_INFRA_CONTAINER_ID=3c607e23e4f51d317f82b2652ba3318dbb8385ee0cc12e1a7618d1f26256aeb9;K8S_POD_UID=130befcf-2f3f-4de4-88c5-6390d88f2b01" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4/130befcf-2f3f-4de4-88c5-6390d88f2b01:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4 3c607e23e4f51d317f82b2652ba3318dbb8385ee0cc12e1a7618d1f26256aeb9 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:25 [10.132.0.37/23] Sep 11 19:38:04.381646 ip-10-0-137-97 kubenswrapper[2559]: ' Sep 11 19:38:04.381646 ip-10-0-137-97 kubenswrapper[2559]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 19:38:04.381646 ip-10-0-137-97 kubenswrapper[2559]: > Sep 11 19:38:04.381997 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.381667 2559 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 11 19:38:04.381997 ip-10-0-137-97 kubenswrapper[2559]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-8zkj4_kuadrant-system_130befcf-2f3f-4de4-88c5-6390d88f2b01_0(3c607e23e4f51d317f82b2652ba3318dbb8385ee0cc12e1a7618d1f26256aeb9): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-8zkj4 to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"3c607e23e4f51d317f82b2652ba3318dbb8385ee0cc12e1a7618d1f26256aeb9" Netns:"/var/run/netns/5d226ea9-8b2a-4980-af3b-6bbc8ab779a5" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-8zkj4;K8S_POD_INFRA_CONTAINER_ID=3c607e23e4f51d317f82b2652ba3318dbb8385ee0cc12e1a7618d1f26256aeb9;K8S_POD_UID=130befcf-2f3f-4de4-88c5-6390d88f2b01" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4/130befcf-2f3f-4de4-88c5-6390d88f2b01:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4 3c607e23e4f51d317f82b2652ba3318dbb8385ee0cc12e1a7618d1f26256aeb9 network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:25 [10.132.0.37/23] Sep 11 19:38:04.381997 ip-10-0-137-97 kubenswrapper[2559]: ' Sep 11 19:38:04.381997 ip-10-0-137-97 kubenswrapper[2559]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 19:38:04.381997 ip-10-0-137-97 kubenswrapper[2559]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:04.381997 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.381758 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51f06832-aba7-4956-8da6-db3b8b20284c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ml6bh\" (UID: \"51f06832-aba7-4956-8da6-db3b8b20284c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:04.387576 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.387558 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:04.390574 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.390551 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:04.392105 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.392090 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:04.396865 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.396840 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t9rsr\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:04.399502 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.399427 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:04.401264 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.401240 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scsjg\" (UniqueName: \"kubernetes.io/projected/51f06832-aba7-4956-8da6-db3b8b20284c-kube-api-access-scsjg\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-ml6bh\" (UID: \"51f06832-aba7-4956-8da6-db3b8b20284c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:04.401848 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:04.401823 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t9rsr\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:04.482541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.482430 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca36913a-a511-4561-8787-e824bf6eb6e7-extensions-socket-volume\") pod \"ca36913a-a511-4561-8787-e824bf6eb6e7\" (UID: \"ca36913a-a511-4561-8787-e824bf6eb6e7\") " Sep 11 19:38:04.482541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.482540 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82kd6\" (UniqueName: \"kubernetes.io/projected/460c8ca1-752e-4cc4-9206-9b705adbdcde-kube-api-access-82kd6\") pod \"460c8ca1-752e-4cc4-9206-9b705adbdcde\" (UID: \"460c8ca1-752e-4cc4-9206-9b705adbdcde\") " Sep 11 19:38:04.482712 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.482599 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jv762\" (UniqueName: \"kubernetes.io/projected/ca36913a-a511-4561-8787-e824bf6eb6e7-kube-api-access-jv762\") pod \"ca36913a-a511-4561-8787-e824bf6eb6e7\" (UID: \"ca36913a-a511-4561-8787-e824bf6eb6e7\") " Sep 11 19:38:04.482796 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.482776 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f2b5\" (UniqueName: \"kubernetes.io/projected/2818e853-46df-4e26-9c31-0f77fdf9c5c8-kube-api-access-6f2b5\") pod \"limitador-operator-controller-manager-74495c69f8-prh59\" (UID: \"2818e853-46df-4e26-9c31-0f77fdf9c5c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" Sep 11 19:38:04.482960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.482934 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca36913a-a511-4561-8787-e824bf6eb6e7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ca36913a-a511-4561-8787-e824bf6eb6e7" (UID: "ca36913a-a511-4561-8787-e824bf6eb6e7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:38:04.484928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.484904 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca36913a-a511-4561-8787-e824bf6eb6e7-kube-api-access-jv762" (OuterVolumeSpecName: "kube-api-access-jv762") pod "ca36913a-a511-4561-8787-e824bf6eb6e7" (UID: "ca36913a-a511-4561-8787-e824bf6eb6e7"). InnerVolumeSpecName "kube-api-access-jv762". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:38:04.484928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.484926 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/460c8ca1-752e-4cc4-9206-9b705adbdcde-kube-api-access-82kd6" (OuterVolumeSpecName: "kube-api-access-82kd6") pod "460c8ca1-752e-4cc4-9206-9b705adbdcde" (UID: "460c8ca1-752e-4cc4-9206-9b705adbdcde"). InnerVolumeSpecName "kube-api-access-82kd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:38:04.502163 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.502134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f2b5\" (UniqueName: \"kubernetes.io/projected/2818e853-46df-4e26-9c31-0f77fdf9c5c8-kube-api-access-6f2b5\") pod \"limitador-operator-controller-manager-74495c69f8-prh59\" (UID: \"2818e853-46df-4e26-9c31-0f77fdf9c5c8\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" Sep 11 19:38:04.536116 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.536091 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:04.545901 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.545882 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" Sep 11 19:38:04.583396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.583355 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv762\" (UniqueName: \"kubernetes.io/projected/ca36913a-a511-4561-8787-e824bf6eb6e7-kube-api-access-jv762\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:38:04.583653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.583399 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ca36913a-a511-4561-8787-e824bf6eb6e7-extensions-socket-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:38:04.583653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.583418 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82kd6\" (UniqueName: \"kubernetes.io/projected/460c8ca1-752e-4cc4-9206-9b705adbdcde-kube-api-access-82kd6\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:38:04.827487 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.827463 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh"] Sep 11 19:38:04.829407 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:38:04.829380 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f06832_aba7_4956_8da6_db3b8b20284c.slice/crio-ec8b4bcb3d29aa3dfb61d42b5f1714d196b2849b9b8a6fa80b344372064ce62a WatchSource:0}: Error finding container ec8b4bcb3d29aa3dfb61d42b5f1714d196b2849b9b8a6fa80b344372064ce62a: Status 404 returned error can't find the container with id ec8b4bcb3d29aa3dfb61d42b5f1714d196b2849b9b8a6fa80b344372064ce62a Sep 11 19:38:04.845045 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:04.845021 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59"] Sep 11 19:38:04.847363 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:38:04.847338 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2818e853_46df_4e26_9c31_0f77fdf9c5c8.slice/crio-e269749f355796657237239b9f1e2c1be0055b90fb73f8e6351874c5432cc699 WatchSource:0}: Error finding container e269749f355796657237239b9f1e2c1be0055b90fb73f8e6351874c5432cc699: Status 404 returned error can't find the container with id e269749f355796657237239b9f1e2c1be0055b90fb73f8e6351874c5432cc699 Sep 11 19:38:05.076892 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.076809 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" event={"ID":"51f06832-aba7-4956-8da6-db3b8b20284c","Type":"ContainerStarted","Data":"3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c"} Sep 11 19:38:05.076892 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.076862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" event={"ID":"51f06832-aba7-4956-8da6-db3b8b20284c","Type":"ContainerStarted","Data":"ec8b4bcb3d29aa3dfb61d42b5f1714d196b2849b9b8a6fa80b344372064ce62a"} Sep 11 19:38:05.077106 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.076989 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:05.078399 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.078378 2559 generic.go:356] "Generic (PLEG): container finished" podID="ca36913a-a511-4561-8787-e824bf6eb6e7" containerID="46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa" exitCode=0 Sep 11 19:38:05.078507 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.078417 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:05.078561 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.078523 2559 scope.go:117] "RemoveContainer" containerID="46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa" Sep 11 19:38:05.079826 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:05.079646 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t9rsr\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:05.079986 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.079967 2559 generic.go:356] "Generic (PLEG): container finished" podID="460c8ca1-752e-4cc4-9206-9b705adbdcde" containerID="23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af" exitCode=0 Sep 11 19:38:05.080050 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.080024 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:05.082073 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.082051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" event={"ID":"2818e853-46df-4e26-9c31-0f77fdf9c5c8","Type":"ContainerStarted","Data":"82e9fdccc111b56d40cf6afaf194487ca681338dc94257212564489a68ecd9f7"} Sep 11 19:38:05.082194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.082096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" event={"ID":"2818e853-46df-4e26-9c31-0f77fdf9c5c8","Type":"ContainerStarted","Data":"e269749f355796657237239b9f1e2c1be0055b90fb73f8e6351874c5432cc699"} Sep 11 19:38:05.082194 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.082055 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:05.082318 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.082204 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" Sep 11 19:38:05.089583 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.089568 2559 scope.go:117] "RemoveContainer" containerID="46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa" Sep 11 19:38:05.089951 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:05.089925 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa\": container with ID starting with 46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa not found: ID does not exist" containerID="46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa" Sep 11 19:38:05.090041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.089961 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa"} err="failed to get container status \"46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa\": rpc error: code = NotFound desc = could not find container \"46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa\": container with ID starting with 46d9cf45ff870222329e43af2f24c4e62e09dfde632bd99e84f9d89714c4e7fa not found: ID does not exist" Sep 11 19:38:05.090041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.089982 2559 scope.go:117] "RemoveContainer" containerID="23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af" Sep 11 19:38:05.090895 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.090875 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:05.098116 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.098097 2559 scope.go:117] "RemoveContainer" containerID="23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af" Sep 11 19:38:05.098367 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:05.098347 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af\": container with ID starting with 23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af not found: ID does not exist" containerID="23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af" Sep 11 19:38:05.098415 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.098374 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af"} err="failed to get container status \"23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af\": rpc error: code = NotFound desc = could not find container \"23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af\": container with ID starting with 23aa988210bb1ac7404cee928327e6098ef5310e6a1c657a27ca38032d7523af not found: ID does not exist" Sep 11 19:38:05.103919 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.103877 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" podStartSLOduration=1.103862167 podStartE2EDuration="1.103862167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:05.101509756 +0000 UTC m=+718.374016957" watchObservedRunningTime="2026-09-11 19:38:05.103862167 +0000 UTC m=+718.376369364" Sep 11 19:38:05.104014 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:05.103928 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:05.106105 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:05.106080 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-rbw4r\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-rbw4r" Sep 11 19:38:05.110307 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:05.110278 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="130befcf-2f3f-4de4-88c5-6390d88f2b01" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:05.134484 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.134413 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" podStartSLOduration=1.134397673 podStartE2EDuration="1.134397673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:05.132537527 +0000 UTC m=+718.405044728" watchObservedRunningTime="2026-09-11 19:38:05.134397673 +0000 UTC m=+718.406904873" Sep 11 19:38:05.136135 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:05.136107 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-t9rsr\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-t9rsr" Sep 11 19:38:05.188456 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.188408 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/130befcf-2f3f-4de4-88c5-6390d88f2b01-extensions-socket-volume\") pod \"130befcf-2f3f-4de4-88c5-6390d88f2b01\" (UID: \"130befcf-2f3f-4de4-88c5-6390d88f2b01\") " Sep 11 19:38:05.188620 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.188503 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-289cf\" (UniqueName: \"kubernetes.io/projected/130befcf-2f3f-4de4-88c5-6390d88f2b01-kube-api-access-289cf\") pod \"130befcf-2f3f-4de4-88c5-6390d88f2b01\" (UID: \"130befcf-2f3f-4de4-88c5-6390d88f2b01\") " Sep 11 19:38:05.188734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.188711 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/130befcf-2f3f-4de4-88c5-6390d88f2b01-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "130befcf-2f3f-4de4-88c5-6390d88f2b01" (UID: "130befcf-2f3f-4de4-88c5-6390d88f2b01"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:38:05.188846 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.188831 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/130befcf-2f3f-4de4-88c5-6390d88f2b01-extensions-socket-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:38:05.190705 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.190680 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130befcf-2f3f-4de4-88c5-6390d88f2b01-kube-api-access-289cf" (OuterVolumeSpecName: "kube-api-access-289cf") pod "130befcf-2f3f-4de4-88c5-6390d88f2b01" (UID: "130befcf-2f3f-4de4-88c5-6390d88f2b01"). InnerVolumeSpecName "kube-api-access-289cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:38:05.229462 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.229414 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130befcf-2f3f-4de4-88c5-6390d88f2b01" path="/var/lib/kubelet/pods/130befcf-2f3f-4de4-88c5-6390d88f2b01/volumes" Sep 11 19:38:05.229740 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.229722 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="460c8ca1-752e-4cc4-9206-9b705adbdcde" path="/var/lib/kubelet/pods/460c8ca1-752e-4cc4-9206-9b705adbdcde/volumes" Sep 11 19:38:05.230102 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.230083 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca36913a-a511-4561-8787-e824bf6eb6e7" path="/var/lib/kubelet/pods/ca36913a-a511-4561-8787-e824bf6eb6e7/volumes" Sep 11 19:38:05.289318 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:05.289285 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-289cf\" (UniqueName: \"kubernetes.io/projected/130befcf-2f3f-4de4-88c5-6390d88f2b01-kube-api-access-289cf\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:38:06.090412 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:06.090383 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:06.096051 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:06.096019 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="130befcf-2f3f-4de4-88c5-6390d88f2b01" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:07.222308 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:07.222267 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-8zkj4\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="130befcf-2f3f-4de4-88c5-6390d88f2b01" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-8zkj4" Sep 11 19:38:16.092562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.092526 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:16.092934 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.092605 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-prh59" Sep 11 19:38:16.373478 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.373426 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg"] Sep 11 19:38:16.380803 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.380781 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:16.396168 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.394248 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg"] Sep 11 19:38:16.486822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.486784 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/480d0f51-3b90-4927-9ab9-258632a66d19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dzzpg\" (UID: \"480d0f51-3b90-4927-9ab9-258632a66d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:16.486995 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.486833 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4ql4\" (UniqueName: \"kubernetes.io/projected/480d0f51-3b90-4927-9ab9-258632a66d19-kube-api-access-w4ql4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dzzpg\" (UID: \"480d0f51-3b90-4927-9ab9-258632a66d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:16.587360 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.587322 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/480d0f51-3b90-4927-9ab9-258632a66d19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dzzpg\" (UID: \"480d0f51-3b90-4927-9ab9-258632a66d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:16.587562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.587374 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4ql4\" (UniqueName: \"kubernetes.io/projected/480d0f51-3b90-4927-9ab9-258632a66d19-kube-api-access-w4ql4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dzzpg\" (UID: \"480d0f51-3b90-4927-9ab9-258632a66d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:16.587763 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.587740 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/480d0f51-3b90-4927-9ab9-258632a66d19-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dzzpg\" (UID: \"480d0f51-3b90-4927-9ab9-258632a66d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:16.601896 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.601865 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4ql4\" (UniqueName: \"kubernetes.io/projected/480d0f51-3b90-4927-9ab9-258632a66d19-kube-api-access-w4ql4\") pod \"kuadrant-operator-controller-manager-5769f99d9f-dzzpg\" (UID: \"480d0f51-3b90-4927-9ab9-258632a66d19\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:16.690895 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.690796 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:16.971490 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:16.971461 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg"] Sep 11 19:38:16.973323 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:38:16.973297 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480d0f51_3b90_4927_9ab9_258632a66d19.slice/crio-050a4880f14560a5bcfa5668e8199221250979be206f53fd55ba55f5b9dd8340 WatchSource:0}: Error finding container 050a4880f14560a5bcfa5668e8199221250979be206f53fd55ba55f5b9dd8340: Status 404 returned error can't find the container with id 050a4880f14560a5bcfa5668e8199221250979be206f53fd55ba55f5b9dd8340 Sep 11 19:38:17.151555 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:17.151521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" event={"ID":"480d0f51-3b90-4927-9ab9-258632a66d19","Type":"ContainerStarted","Data":"c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1"} Sep 11 19:38:17.151945 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:17.151575 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" event={"ID":"480d0f51-3b90-4927-9ab9-258632a66d19","Type":"ContainerStarted","Data":"050a4880f14560a5bcfa5668e8199221250979be206f53fd55ba55f5b9dd8340"} Sep 11 19:38:17.151945 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:17.151663 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:17.189503 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:17.189428 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" podStartSLOduration=1.189411164 podStartE2EDuration="1.189411164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:17.187328759 +0000 UTC m=+730.459835958" watchObservedRunningTime="2026-09-11 19:38:17.189411164 +0000 UTC m=+730.461918364" Sep 11 19:38:28.159513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.159471 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 19:38:28.221036 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.221001 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh"] Sep 11 19:38:28.221254 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.221229 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" podUID="51f06832-aba7-4956-8da6-db3b8b20284c" containerName="manager" containerID="cri-o://3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c" gracePeriod=10 Sep 11 19:38:28.467561 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.467538 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:28.486907 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.486880 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scsjg\" (UniqueName: \"kubernetes.io/projected/51f06832-aba7-4956-8da6-db3b8b20284c-kube-api-access-scsjg\") pod \"51f06832-aba7-4956-8da6-db3b8b20284c\" (UID: \"51f06832-aba7-4956-8da6-db3b8b20284c\") " Sep 11 19:38:28.487058 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.487020 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51f06832-aba7-4956-8da6-db3b8b20284c-extensions-socket-volume\") pod \"51f06832-aba7-4956-8da6-db3b8b20284c\" (UID: \"51f06832-aba7-4956-8da6-db3b8b20284c\") " Sep 11 19:38:28.487608 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.487499 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51f06832-aba7-4956-8da6-db3b8b20284c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "51f06832-aba7-4956-8da6-db3b8b20284c" (UID: "51f06832-aba7-4956-8da6-db3b8b20284c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:38:28.489379 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.489350 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f06832-aba7-4956-8da6-db3b8b20284c-kube-api-access-scsjg" (OuterVolumeSpecName: "kube-api-access-scsjg") pod "51f06832-aba7-4956-8da6-db3b8b20284c" (UID: "51f06832-aba7-4956-8da6-db3b8b20284c"). InnerVolumeSpecName "kube-api-access-scsjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:38:28.588581 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.588542 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/51f06832-aba7-4956-8da6-db3b8b20284c-extensions-socket-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:38:28.588581 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:28.588572 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scsjg\" (UniqueName: \"kubernetes.io/projected/51f06832-aba7-4956-8da6-db3b8b20284c-kube-api-access-scsjg\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:38:29.214375 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.214336 2559 generic.go:356] "Generic (PLEG): container finished" podID="51f06832-aba7-4956-8da6-db3b8b20284c" containerID="3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c" exitCode=0 Sep 11 19:38:29.214849 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.214388 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" event={"ID":"51f06832-aba7-4956-8da6-db3b8b20284c","Type":"ContainerDied","Data":"3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c"} Sep 11 19:38:29.214849 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.214394 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" Sep 11 19:38:29.214849 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.214479 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh" event={"ID":"51f06832-aba7-4956-8da6-db3b8b20284c","Type":"ContainerDied","Data":"ec8b4bcb3d29aa3dfb61d42b5f1714d196b2849b9b8a6fa80b344372064ce62a"} Sep 11 19:38:29.214849 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.214502 2559 scope.go:117] "RemoveContainer" containerID="3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c" Sep 11 19:38:29.225019 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.224902 2559 scope.go:117] "RemoveContainer" containerID="3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c" Sep 11 19:38:29.225198 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:29.225178 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c\": container with ID starting with 3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c not found: ID does not exist" containerID="3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c" Sep 11 19:38:29.225269 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.225209 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c"} err="failed to get container status \"3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c\": rpc error: code = NotFound desc = could not find container \"3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c\": container with ID starting with 3ecf2cf165dfca4e237d678821dc34648177d20c6a6d203ea0be1026b3546b3c not found: ID does not exist" Sep 11 19:38:29.250885 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.250862 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh"] Sep 11 19:38:29.257195 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:29.257173 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-ml6bh"] Sep 11 19:38:31.223670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:31.223639 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f06832-aba7-4956-8da6-db3b8b20284c" path="/var/lib/kubelet/pods/51f06832-aba7-4956-8da6-db3b8b20284c/volumes" Sep 11 19:38:46.733518 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.733481 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:38:46.734031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.734010 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f06832-aba7-4956-8da6-db3b8b20284c" containerName="manager" Sep 11 19:38:46.734121 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.734035 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f06832-aba7-4956-8da6-db3b8b20284c" containerName="manager" Sep 11 19:38:46.734121 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.734120 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f06832-aba7-4956-8da6-db3b8b20284c" containerName="manager" Sep 11 19:38:46.753338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.753310 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:38:46.753503 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.753405 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:46.755869 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.755839 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Sep 11 19:38:46.769457 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.769400 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:38:46.844687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.844648 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wksq8\" (UniqueName: \"kubernetes.io/projected/90c02666-75a6-48b9-9497-5e098f391ce5-kube-api-access-wksq8\") pod \"limitador-limitador-84994f7ffc-srsnl\" (UID: \"90c02666-75a6-48b9-9497-5e098f391ce5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:46.844860 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.844710 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90c02666-75a6-48b9-9497-5e098f391ce5-config-file\") pod \"limitador-limitador-84994f7ffc-srsnl\" (UID: \"90c02666-75a6-48b9-9497-5e098f391ce5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:46.945428 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.945390 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wksq8\" (UniqueName: \"kubernetes.io/projected/90c02666-75a6-48b9-9497-5e098f391ce5-kube-api-access-wksq8\") pod \"limitador-limitador-84994f7ffc-srsnl\" (UID: \"90c02666-75a6-48b9-9497-5e098f391ce5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:46.945649 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.945491 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90c02666-75a6-48b9-9497-5e098f391ce5-config-file\") pod \"limitador-limitador-84994f7ffc-srsnl\" (UID: \"90c02666-75a6-48b9-9497-5e098f391ce5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:46.946155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.946127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/90c02666-75a6-48b9-9497-5e098f391ce5-config-file\") pod \"limitador-limitador-84994f7ffc-srsnl\" (UID: \"90c02666-75a6-48b9-9497-5e098f391ce5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:46.961841 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:46.961804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wksq8\" (UniqueName: \"kubernetes.io/projected/90c02666-75a6-48b9-9497-5e098f391ce5-kube-api-access-wksq8\") pod \"limitador-limitador-84994f7ffc-srsnl\" (UID: \"90c02666-75a6-48b9-9497-5e098f391ce5\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:47.064021 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.063983 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:47.230647 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.230617 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vw97g"] Sep 11 19:38:47.235305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.235283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" Sep 11 19:38:47.238406 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.238382 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v465p\"" Sep 11 19:38:47.247625 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.247600 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vw97g"] Sep 11 19:38:47.346221 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.346114 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:38:47.349160 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.349133 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b8g4\" (UniqueName: \"kubernetes.io/projected/7f427f00-0dd8-454c-b532-c0bb9de324a2-kube-api-access-6b8g4\") pod \"authorino-7bcfd8f9cd-vw97g\" (UID: \"7f427f00-0dd8-454c-b532-c0bb9de324a2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" Sep 11 19:38:47.349274 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:38:47.349172 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c02666_75a6_48b9_9497_5e098f391ce5.slice/crio-3eed37a54f3700e1735555ebff38fd534dd7f112708cfc6e2177ee1aa87fbc35 WatchSource:0}: Error finding container 3eed37a54f3700e1735555ebff38fd534dd7f112708cfc6e2177ee1aa87fbc35: Status 404 returned error can't find the container with id 3eed37a54f3700e1735555ebff38fd534dd7f112708cfc6e2177ee1aa87fbc35 Sep 11 19:38:47.450074 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.450039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b8g4\" (UniqueName: \"kubernetes.io/projected/7f427f00-0dd8-454c-b532-c0bb9de324a2-kube-api-access-6b8g4\") pod \"authorino-7bcfd8f9cd-vw97g\" (UID: \"7f427f00-0dd8-454c-b532-c0bb9de324a2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" Sep 11 19:38:47.458550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.458519 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b8g4\" (UniqueName: \"kubernetes.io/projected/7f427f00-0dd8-454c-b532-c0bb9de324a2-kube-api-access-6b8g4\") pod \"authorino-7bcfd8f9cd-vw97g\" (UID: \"7f427f00-0dd8-454c-b532-c0bb9de324a2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" Sep 11 19:38:47.545114 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.545081 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" Sep 11 19:38:47.814856 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:47.814829 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vw97g"] Sep 11 19:38:47.816491 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:38:47.816467 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f427f00_0dd8_454c_b532_c0bb9de324a2.slice/crio-91b3506ed2039c9a8b54074785970aab26b4080ee33bb341bf060155c1646758 WatchSource:0}: Error finding container 91b3506ed2039c9a8b54074785970aab26b4080ee33bb341bf060155c1646758: Status 404 returned error can't find the container with id 91b3506ed2039c9a8b54074785970aab26b4080ee33bb341bf060155c1646758 Sep 11 19:38:48.321664 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:48.321640 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" event={"ID":"90c02666-75a6-48b9-9497-5e098f391ce5","Type":"ContainerStarted","Data":"3eed37a54f3700e1735555ebff38fd534dd7f112708cfc6e2177ee1aa87fbc35"} Sep 11 19:38:48.322875 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:48.322852 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" event={"ID":"7f427f00-0dd8-454c-b532-c0bb9de324a2","Type":"ContainerStarted","Data":"91b3506ed2039c9a8b54074785970aab26b4080ee33bb341bf060155c1646758"} Sep 11 19:38:49.331109 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:49.331072 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" event={"ID":"90c02666-75a6-48b9-9497-5e098f391ce5","Type":"ContainerStarted","Data":"c58b52e7e7434f4a124554ec047ae3dde6c9bcad6d5eaee6790ac1d67990f56e"} Sep 11 19:38:49.331710 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:49.331245 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:38:49.350100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:49.350043 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" podStartSLOduration=2.386125168 podStartE2EDuration="3.350027383s" totalImagesPullingTime="963.902215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:49.349097825 +0000 UTC m=+762.621605027" watchObservedRunningTime="2026-09-11 19:38:49.350027383 +0000 UTC m=+762.622534582" Sep 11 19:38:49.763200 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:49.763170 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vw97g"] Sep 11 19:38:50.337953 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:50.337910 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" event={"ID":"7f427f00-0dd8-454c-b532-c0bb9de324a2","Type":"ContainerStarted","Data":"99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22"} Sep 11 19:38:50.353416 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:50.353358 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" podStartSLOduration=1.6906921700000002 podStartE2EDuration="3.353338969s" totalImagesPullingTime="1.662646799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:38:50.352838019 +0000 UTC m=+763.625345220" watchObservedRunningTime="2026-09-11 19:38:50.353338969 +0000 UTC m=+763.625846172" Sep 11 19:38:51.343554 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:51.343513 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" podUID="7f427f00-0dd8-454c-b532-c0bb9de324a2" containerName="authorino" containerID="cri-o://99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22" gracePeriod=30 Sep 11 19:38:51.588243 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:51.588209 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" Sep 11 19:38:51.688947 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:51.688868 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6b8g4\" (UniqueName: \"kubernetes.io/projected/7f427f00-0dd8-454c-b532-c0bb9de324a2-kube-api-access-6b8g4\") pod \"7f427f00-0dd8-454c-b532-c0bb9de324a2\" (UID: \"7f427f00-0dd8-454c-b532-c0bb9de324a2\") " Sep 11 19:38:51.691062 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:51.691029 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f427f00-0dd8-454c-b532-c0bb9de324a2-kube-api-access-6b8g4" (OuterVolumeSpecName: "kube-api-access-6b8g4") pod "7f427f00-0dd8-454c-b532-c0bb9de324a2" (UID: "7f427f00-0dd8-454c-b532-c0bb9de324a2"). InnerVolumeSpecName "kube-api-access-6b8g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:38:51.789790 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:51.789757 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b8g4\" (UniqueName: \"kubernetes.io/projected/7f427f00-0dd8-454c-b532-c0bb9de324a2-kube-api-access-6b8g4\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:38:52.350272 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.350241 2559 generic.go:356] "Generic (PLEG): container finished" podID="7f427f00-0dd8-454c-b532-c0bb9de324a2" containerID="99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22" exitCode=0 Sep 11 19:38:52.350791 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.350299 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" Sep 11 19:38:52.350791 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.350302 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" event={"ID":"7f427f00-0dd8-454c-b532-c0bb9de324a2","Type":"ContainerDied","Data":"99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22"} Sep 11 19:38:52.350791 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.350406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-vw97g" event={"ID":"7f427f00-0dd8-454c-b532-c0bb9de324a2","Type":"ContainerDied","Data":"91b3506ed2039c9a8b54074785970aab26b4080ee33bb341bf060155c1646758"} Sep 11 19:38:52.350791 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.350422 2559 scope.go:117] "RemoveContainer" containerID="99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22" Sep 11 19:38:52.359513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.359497 2559 scope.go:117] "RemoveContainer" containerID="99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22" Sep 11 19:38:52.359753 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:38:52.359737 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22\": container with ID starting with 99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22 not found: ID does not exist" containerID="99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22" Sep 11 19:38:52.359818 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.359760 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22"} err="failed to get container status \"99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22\": rpc error: code = NotFound desc = could not find container \"99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22\": container with ID starting with 99476c2dd2e7f156199d668c9c4b48a45971718f5ea27ddd1aed26a6feffeb22 not found: ID does not exist" Sep 11 19:38:52.370751 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.370730 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vw97g"] Sep 11 19:38:52.374849 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:52.374826 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-vw97g"] Sep 11 19:38:53.222870 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:38:53.222827 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f427f00-0dd8-454c-b532-c0bb9de324a2" path="/var/lib/kubelet/pods/7f427f00-0dd8-454c-b532-c0bb9de324a2/volumes" Sep 11 19:39:00.338660 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:00.338631 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-srsnl" Sep 11 19:39:14.469090 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:14.469056 2559 generic.go:356] "Generic (PLEG): container finished" podID="1e2c2640-d759-41bc-bc41-7f57b39cb976" containerID="6f3ce508cab7e3af2ea688dda56778745945655609b8b2bba4cba0eddc8b49f3" exitCode=1 Sep 11 19:39:14.469588 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:14.469134 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" event={"ID":"1e2c2640-d759-41bc-bc41-7f57b39cb976","Type":"ContainerDied","Data":"6f3ce508cab7e3af2ea688dda56778745945655609b8b2bba4cba0eddc8b49f3"} Sep 11 19:39:14.469655 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:14.469613 2559 scope.go:117] "RemoveContainer" containerID="6f3ce508cab7e3af2ea688dda56778745945655609b8b2bba4cba0eddc8b49f3" Sep 11 19:39:14.543548 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:14.543516 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:39:15.047009 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:15.046969 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:39:15.477130 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:15.477092 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" event={"ID":"1e2c2640-d759-41bc-bc41-7f57b39cb976","Type":"ContainerStarted","Data":"63a2dcecf4bc56a5a98c247597a7eb002970b36dce9eee39bec334d53e352898"} Sep 11 19:39:15.477583 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:15.477190 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:39:15.482051 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:15.482031 2559 generic.go:356] "Generic (PLEG): container finished" podID="d4bffeba-f3e3-49a1-95c6-31b4a31d0755" containerID="252540d08ecf0701094d43bc4d3cad8a4d9e4801c202943ccbc35cd57dcfc808" exitCode=1 Sep 11 19:39:15.482180 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:15.482117 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" event={"ID":"d4bffeba-f3e3-49a1-95c6-31b4a31d0755","Type":"ContainerDied","Data":"252540d08ecf0701094d43bc4d3cad8a4d9e4801c202943ccbc35cd57dcfc808"} Sep 11 19:39:15.482547 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:15.482533 2559 scope.go:117] "RemoveContainer" containerID="252540d08ecf0701094d43bc4d3cad8a4d9e4801c202943ccbc35cd57dcfc808" Sep 11 19:39:16.489303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:16.489272 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" event={"ID":"d4bffeba-f3e3-49a1-95c6-31b4a31d0755","Type":"ContainerStarted","Data":"4520e6b679f2c2b9bcdb1bf7ae8a7b99fd5eef6595241691071df7c98459ce36"} Sep 11 19:39:16.489734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:16.489658 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:39:27.496565 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:27.496529 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c47b89ccd-kgkdj" Sep 11 19:39:46.491093 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:39:46.491064 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-88bmh" Sep 11 19:40:01.572121 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.572088 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-42vs8"] Sep 11 19:40:01.572733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.572492 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f427f00-0dd8-454c-b532-c0bb9de324a2" containerName="authorino" Sep 11 19:40:01.572733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.572505 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f427f00-0dd8-454c-b532-c0bb9de324a2" containerName="authorino" Sep 11 19:40:01.572733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.572579 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f427f00-0dd8-454c-b532-c0bb9de324a2" containerName="authorino" Sep 11 19:40:01.575516 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.575497 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-42vs8" Sep 11 19:40:01.579833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.579812 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-v465p\"" Sep 11 19:40:01.586541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.586505 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-42vs8"] Sep 11 19:40:01.682523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.682476 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f9qj\" (UniqueName: \"kubernetes.io/projected/dabe009a-c395-46f3-af6c-031612570b07-kube-api-access-8f9qj\") pod \"authorino-557f876f69-42vs8\" (UID: \"dabe009a-c395-46f3-af6c-031612570b07\") " pod="kuadrant-system/authorino-557f876f69-42vs8" Sep 11 19:40:01.691529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.691496 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-42vs8"] Sep 11 19:40:01.691765 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:40:01.691744 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-8f9qj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-42vs8" podUID="dabe009a-c395-46f3-af6c-031612570b07" Sep 11 19:40:01.736747 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.736707 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-42vs8" Sep 11 19:40:01.743556 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.743532 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-42vs8" Sep 11 19:40:01.783307 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.783268 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f9qj\" (UniqueName: \"kubernetes.io/projected/dabe009a-c395-46f3-af6c-031612570b07-kube-api-access-8f9qj\") pod \"authorino-557f876f69-42vs8\" (UID: \"dabe009a-c395-46f3-af6c-031612570b07\") " pod="kuadrant-system/authorino-557f876f69-42vs8" Sep 11 19:40:01.793063 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.793031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f9qj\" (UniqueName: \"kubernetes.io/projected/dabe009a-c395-46f3-af6c-031612570b07-kube-api-access-8f9qj\") pod \"authorino-557f876f69-42vs8\" (UID: \"dabe009a-c395-46f3-af6c-031612570b07\") " pod="kuadrant-system/authorino-557f876f69-42vs8" Sep 11 19:40:01.831767 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.831692 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6cf998cbb6-qz7rf"] Sep 11 19:40:01.835122 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.835103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:40:01.837902 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.837884 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 19:40:01.844078 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.844052 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf998cbb6-qz7rf"] Sep 11 19:40:01.984788 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.984753 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f9qj\" (UniqueName: \"kubernetes.io/projected/dabe009a-c395-46f3-af6c-031612570b07-kube-api-access-8f9qj\") pod \"dabe009a-c395-46f3-af6c-031612570b07\" (UID: \"dabe009a-c395-46f3-af6c-031612570b07\") " Sep 11 19:40:01.984959 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.984909 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2rjr\" (UniqueName: \"kubernetes.io/projected/00a5baf1-58a5-458a-b052-05ec33c00b7f-kube-api-access-w2rjr\") pod \"authorino-6cf998cbb6-qz7rf\" (UID: \"00a5baf1-58a5-458a-b052-05ec33c00b7f\") " pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:40:01.985004 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.984964 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00a5baf1-58a5-458a-b052-05ec33c00b7f-tls-cert\") pod \"authorino-6cf998cbb6-qz7rf\" (UID: \"00a5baf1-58a5-458a-b052-05ec33c00b7f\") " pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:40:01.987045 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:01.987017 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dabe009a-c395-46f3-af6c-031612570b07-kube-api-access-8f9qj" (OuterVolumeSpecName: "kube-api-access-8f9qj") pod "dabe009a-c395-46f3-af6c-031612570b07" (UID: "dabe009a-c395-46f3-af6c-031612570b07"). InnerVolumeSpecName "kube-api-access-8f9qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:40:02.085542 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.085427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00a5baf1-58a5-458a-b052-05ec33c00b7f-tls-cert\") pod \"authorino-6cf998cbb6-qz7rf\" (UID: \"00a5baf1-58a5-458a-b052-05ec33c00b7f\") " pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:40:02.085542 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.085533 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2rjr\" (UniqueName: \"kubernetes.io/projected/00a5baf1-58a5-458a-b052-05ec33c00b7f-kube-api-access-w2rjr\") pod \"authorino-6cf998cbb6-qz7rf\" (UID: \"00a5baf1-58a5-458a-b052-05ec33c00b7f\") " pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:40:02.085721 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.085612 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f9qj\" (UniqueName: \"kubernetes.io/projected/dabe009a-c395-46f3-af6c-031612570b07-kube-api-access-8f9qj\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:40:02.087966 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.087937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00a5baf1-58a5-458a-b052-05ec33c00b7f-tls-cert\") pod \"authorino-6cf998cbb6-qz7rf\" (UID: \"00a5baf1-58a5-458a-b052-05ec33c00b7f\") " pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:40:02.095021 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.094998 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2rjr\" (UniqueName: \"kubernetes.io/projected/00a5baf1-58a5-458a-b052-05ec33c00b7f-kube-api-access-w2rjr\") pod \"authorino-6cf998cbb6-qz7rf\" (UID: \"00a5baf1-58a5-458a-b052-05ec33c00b7f\") " pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:40:02.145985 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.145934 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:40:02.419604 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.419579 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6cf998cbb6-qz7rf"] Sep 11 19:40:02.421286 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:40:02.421247 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00a5baf1_58a5_458a_b052_05ec33c00b7f.slice/crio-0273affa96f98d257b40e73f03fd01b537d4c48c8cd6ce55c1ec74f25b2c015e WatchSource:0}: Error finding container 0273affa96f98d257b40e73f03fd01b537d4c48c8cd6ce55c1ec74f25b2c015e: Status 404 returned error can't find the container with id 0273affa96f98d257b40e73f03fd01b537d4c48c8cd6ce55c1ec74f25b2c015e Sep 11 19:40:02.743192 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.743143 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" event={"ID":"00a5baf1-58a5-458a-b052-05ec33c00b7f","Type":"ContainerStarted","Data":"0273affa96f98d257b40e73f03fd01b537d4c48c8cd6ce55c1ec74f25b2c015e"} Sep 11 19:40:02.743192 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.743174 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-42vs8" Sep 11 19:40:02.779459 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.779399 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-42vs8"] Sep 11 19:40:02.780656 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:02.780633 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-42vs8"] Sep 11 19:40:03.222978 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:03.222902 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dabe009a-c395-46f3-af6c-031612570b07" path="/var/lib/kubelet/pods/dabe009a-c395-46f3-af6c-031612570b07/volumes" Sep 11 19:40:13.801977 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:13.801940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" event={"ID":"00a5baf1-58a5-458a-b052-05ec33c00b7f","Type":"ContainerStarted","Data":"7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6"} Sep 11 19:40:13.818713 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:13.818657 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" podStartSLOduration=2.193112568 podStartE2EDuration="12.818640902s" totalImagesPullingTime="10.625528334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:40:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:40:13.816747319 +0000 UTC m=+847.089254522" watchObservedRunningTime="2026-09-11 19:40:13.818640902 +0000 UTC m=+847.091148102" Sep 11 19:40:59.059395 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:40:59.059316 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:41:28.003490 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:28.003454 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:41:44.673048 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:44.673008 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:41:50.766505 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.766472 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p"] Sep 11 19:41:50.769839 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.769813 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.774553 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.774522 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 19:41:50.774553 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.774531 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 19:41:50.774753 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.774592 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 19:41:50.774753 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.774531 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-cm7wn\"" Sep 11 19:41:50.783591 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.783566 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p"] Sep 11 19:41:50.890050 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.889998 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.890230 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.890068 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.890230 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.890147 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.890230 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.890207 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.890338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.890268 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.890338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.890297 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8dk5\" (UniqueName: \"kubernetes.io/projected/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-kube-api-access-n8dk5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991098 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991104 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991137 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8dk5\" (UniqueName: \"kubernetes.io/projected/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-kube-api-access-n8dk5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991172 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991227 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991271 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991615 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991544 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991615 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991582 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.991699 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.991647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.993687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.993665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:50.993935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:50.993917 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:51.002227 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:51.002205 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8dk5\" (UniqueName: \"kubernetes.io/projected/d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1-kube-api-access-n8dk5\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p\" (UID: \"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:51.079575 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:51.079488 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:41:51.355621 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:51.355592 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p"] Sep 11 19:41:51.360667 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:51.360633 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 19:41:51.777716 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:51.777684 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:41:52.325509 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:52.325465 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" event={"ID":"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1","Type":"ContainerStarted","Data":"6752281a7926c847555c21e29dd162f4e5507e3fcc0fd89ebe02020ea21ddd38"} Sep 11 19:41:53.332103 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:53.332070 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" event={"ID":"d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1","Type":"ContainerStarted","Data":"43a3eb1e142ed91b98074d2ce92420e3c090b4173ec0d27745d0e82e604ee409"} Sep 11 19:41:53.356301 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:53.356249 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" podStartSLOduration=2.306819204 podStartE2EDuration="3.356235125s" totalImagesPullingTime="1.049415921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:41:53.354169043 +0000 UTC m=+946.626676243" watchObservedRunningTime="2026-09-11 19:41:53.356235125 +0000 UTC m=+946.628742324" Sep 11 19:41:56.912397 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:56.912361 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45"] Sep 11 19:41:56.915877 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:56.915861 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:56.924279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:56.924260 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 19:41:56.939346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:56.939317 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45"] Sep 11 19:41:57.049180 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.049145 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.049351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.049211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.049351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.049262 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.049351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.049295 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10aed8d0-1812-452b-8bea-013d57318766-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.049351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.049317 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.049351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.049333 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp7f4\" (UniqueName: \"kubernetes.io/projected/10aed8d0-1812-452b-8bea-013d57318766-kube-api-access-zp7f4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150111 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150072 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150152 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10aed8d0-1812-452b-8bea-013d57318766-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150197 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp7f4\" (UniqueName: \"kubernetes.io/projected/10aed8d0-1812-452b-8bea-013d57318766-kube-api-access-zp7f4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150636 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150546 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150681 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150628 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.150717 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.150681 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.152649 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.152623 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10aed8d0-1812-452b-8bea-013d57318766-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.152953 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.152936 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10aed8d0-1812-452b-8bea-013d57318766-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.163072 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.163020 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp7f4\" (UniqueName: \"kubernetes.io/projected/10aed8d0-1812-452b-8bea-013d57318766-kube-api-access-zp7f4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7rj45\" (UID: \"10aed8d0-1812-452b-8bea-013d57318766\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.227876 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.227845 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:41:57.508297 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.508270 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45"] Sep 11 19:41:57.510061 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:41:57.510034 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10aed8d0_1812_452b_8bea_013d57318766.slice/crio-d88e35eb6ac66a465241352b3fc99855c9aaac162e5e9bd5bba3f010b52948c9 WatchSource:0}: Error finding container d88e35eb6ac66a465241352b3fc99855c9aaac162e5e9bd5bba3f010b52948c9: Status 404 returned error can't find the container with id d88e35eb6ac66a465241352b3fc99855c9aaac162e5e9bd5bba3f010b52948c9 Sep 11 19:41:57.669091 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:57.669060 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:41:58.361852 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:58.361803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" event={"ID":"10aed8d0-1812-452b-8bea-013d57318766","Type":"ContainerStarted","Data":"6fb17472652a74b05f47c1fe3e34099791ccad8d0e45708909c802e56869780d"} Sep 11 19:41:58.362260 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:58.361866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" event={"ID":"10aed8d0-1812-452b-8bea-013d57318766","Type":"ContainerStarted","Data":"d88e35eb6ac66a465241352b3fc99855c9aaac162e5e9bd5bba3f010b52948c9"} Sep 11 19:41:58.388873 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:41:58.388820 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" podStartSLOduration=2.2005046249999998 podStartE2EDuration="2.388798493s" totalImagesPullingTime="188.293868ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:41:58.38819054 +0000 UTC m=+951.660697742" watchObservedRunningTime="2026-09-11 19:41:58.388798493 +0000 UTC m=+951.661305694" Sep 11 19:42:01.080426 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:01.080392 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:42:01.080821 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:01.080475 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:42:01.092645 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:01.092618 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:42:01.390694 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:01.390610 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p" Sep 11 19:42:07.228780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:07.228751 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:42:07.229239 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:07.229104 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:42:07.240984 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:07.240953 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:42:07.424396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:07.424364 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7rj45" Sep 11 19:42:13.483731 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:13.483694 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:42:26.963314 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:26.963277 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:42:29.558131 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.558049 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n"] Sep 11 19:42:29.564638 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.564613 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.567302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.567282 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 11 19:42:29.572337 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.572319 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n"] Sep 11 19:42:29.624564 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.624520 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.624564 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.624562 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.624822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.624696 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.624822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.624736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47wn\" (UniqueName: \"kubernetes.io/projected/5213e0e1-9121-4c72-a026-4f280dda8a5c-kube-api-access-b47wn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.624822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.624804 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5213e0e1-9121-4c72-a026-4f280dda8a5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.624987 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.624843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725246 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725212 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5213e0e1-9121-4c72-a026-4f280dda8a5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725246 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725249 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725296 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725378 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725404 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47wn\" (UniqueName: \"kubernetes.io/projected/5213e0e1-9121-4c72-a026-4f280dda8a5c-kube-api-access-b47wn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725809 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725976 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725854 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.725976 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.725867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.727881 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.727849 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5213e0e1-9121-4c72-a026-4f280dda8a5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.728024 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.728004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5213e0e1-9121-4c72-a026-4f280dda8a5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.740995 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.740965 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47wn\" (UniqueName: \"kubernetes.io/projected/5213e0e1-9121-4c72-a026-4f280dda8a5c-kube-api-access-b47wn\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n\" (UID: \"5213e0e1-9121-4c72-a026-4f280dda8a5c\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:29.874502 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:29.874387 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:30.150290 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:30.150263 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n"] Sep 11 19:42:30.152773 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:42:30.152741 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5213e0e1_9121_4c72_a026_4f280dda8a5c.slice/crio-b7ea0d3aa08d8364b9e5463bacc2d10970eef8ac97a747d0acfda31c192ee71e WatchSource:0}: Error finding container b7ea0d3aa08d8364b9e5463bacc2d10970eef8ac97a747d0acfda31c192ee71e: Status 404 returned error can't find the container with id b7ea0d3aa08d8364b9e5463bacc2d10970eef8ac97a747d0acfda31c192ee71e Sep 11 19:42:30.544141 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:30.544107 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" event={"ID":"5213e0e1-9121-4c72-a026-4f280dda8a5c","Type":"ContainerStarted","Data":"43ca6d444a351b20f9913769c122de92e2d03243630db7fed5998a512505e518"} Sep 11 19:42:30.544141 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:30.544151 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" event={"ID":"5213e0e1-9121-4c72-a026-4f280dda8a5c","Type":"ContainerStarted","Data":"b7ea0d3aa08d8364b9e5463bacc2d10970eef8ac97a747d0acfda31c192ee71e"} Sep 11 19:42:30.566067 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:30.566012 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" podStartSLOduration=1.387222899 podStartE2EDuration="1.565992704s" totalImagesPullingTime="178.769805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:42:30.564385258 +0000 UTC m=+983.836892458" watchObservedRunningTime="2026-09-11 19:42:30.565992704 +0000 UTC m=+983.838499938" Sep 11 19:42:36.060340 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:36.060297 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:42:39.875578 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:39.875543 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:39.875949 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:39.875596 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:39.887792 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:39.887765 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:42:40.614331 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:42:40.614302 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n" Sep 11 19:44:28.259741 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.259702 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-b4d6bd67f-545jz"] Sep 11 19:44:28.263131 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.263108 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4d6bd67f-545jz" Sep 11 19:44:28.285921 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.285887 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b4d6bd67f-545jz"] Sep 11 19:44:28.286691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.286664 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d68df59-dab0-4bf2-bb4e-80f28f7797cb-tls-cert\") pod \"authorino-b4d6bd67f-545jz\" (UID: \"9d68df59-dab0-4bf2-bb4e-80f28f7797cb\") " pod="kuadrant-system/authorino-b4d6bd67f-545jz" Sep 11 19:44:28.286811 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.286736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjp4\" (UniqueName: \"kubernetes.io/projected/9d68df59-dab0-4bf2-bb4e-80f28f7797cb-kube-api-access-nmjp4\") pod \"authorino-b4d6bd67f-545jz\" (UID: \"9d68df59-dab0-4bf2-bb4e-80f28f7797cb\") " pod="kuadrant-system/authorino-b4d6bd67f-545jz" Sep 11 19:44:28.388142 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.388103 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d68df59-dab0-4bf2-bb4e-80f28f7797cb-tls-cert\") pod \"authorino-b4d6bd67f-545jz\" (UID: \"9d68df59-dab0-4bf2-bb4e-80f28f7797cb\") " pod="kuadrant-system/authorino-b4d6bd67f-545jz" Sep 11 19:44:28.388312 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.388168 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjp4\" (UniqueName: \"kubernetes.io/projected/9d68df59-dab0-4bf2-bb4e-80f28f7797cb-kube-api-access-nmjp4\") pod \"authorino-b4d6bd67f-545jz\" (UID: \"9d68df59-dab0-4bf2-bb4e-80f28f7797cb\") " pod="kuadrant-system/authorino-b4d6bd67f-545jz" Sep 11 19:44:28.390780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.390759 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/9d68df59-dab0-4bf2-bb4e-80f28f7797cb-tls-cert\") pod \"authorino-b4d6bd67f-545jz\" (UID: \"9d68df59-dab0-4bf2-bb4e-80f28f7797cb\") " pod="kuadrant-system/authorino-b4d6bd67f-545jz" Sep 11 19:44:28.397508 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.397488 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjp4\" (UniqueName: \"kubernetes.io/projected/9d68df59-dab0-4bf2-bb4e-80f28f7797cb-kube-api-access-nmjp4\") pod \"authorino-b4d6bd67f-545jz\" (UID: \"9d68df59-dab0-4bf2-bb4e-80f28f7797cb\") " pod="kuadrant-system/authorino-b4d6bd67f-545jz" Sep 11 19:44:28.571740 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.571650 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-b4d6bd67f-545jz" Sep 11 19:44:28.850057 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:28.850027 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-b4d6bd67f-545jz"] Sep 11 19:44:28.851988 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:44:28.851962 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d68df59_dab0_4bf2_bb4e_80f28f7797cb.slice/crio-a4d5c2041341e317e2c0ea4990112df7276f36e3f066ce0b795a8dcca7b57426 WatchSource:0}: Error finding container a4d5c2041341e317e2c0ea4990112df7276f36e3f066ce0b795a8dcca7b57426: Status 404 returned error can't find the container with id a4d5c2041341e317e2c0ea4990112df7276f36e3f066ce0b795a8dcca7b57426 Sep 11 19:44:29.198046 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:29.197958 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4d6bd67f-545jz" event={"ID":"9d68df59-dab0-4bf2-bb4e-80f28f7797cb","Type":"ContainerStarted","Data":"a4d5c2041341e317e2c0ea4990112df7276f36e3f066ce0b795a8dcca7b57426"} Sep 11 19:44:30.207085 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.207035 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-b4d6bd67f-545jz" event={"ID":"9d68df59-dab0-4bf2-bb4e-80f28f7797cb","Type":"ContainerStarted","Data":"29153744e9c40158f31a1170a02802f9ba2baba1933e5496f1bd025d729b793b"} Sep 11 19:44:30.230837 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.230780 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-b4d6bd67f-545jz" podStartSLOduration=1.407439039 podStartE2EDuration="2.230762821s" totalImagesPullingTime="823.323782ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:30.228143671 +0000 UTC m=+1103.500650872" watchObservedRunningTime="2026-09-11 19:44:30.230762821 +0000 UTC m=+1103.503270055" Sep 11 19:44:30.267068 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.267033 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cf998cbb6-qz7rf"] Sep 11 19:44:30.267282 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.267254 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" podUID="00a5baf1-58a5-458a-b052-05ec33c00b7f" containerName="authorino" containerID="cri-o://7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6" gracePeriod=30 Sep 11 19:44:30.537095 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.536818 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:44:30.602488 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.602420 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00a5baf1-58a5-458a-b052-05ec33c00b7f-tls-cert\") pod \"00a5baf1-58a5-458a-b052-05ec33c00b7f\" (UID: \"00a5baf1-58a5-458a-b052-05ec33c00b7f\") " Sep 11 19:44:30.602687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.602533 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2rjr\" (UniqueName: \"kubernetes.io/projected/00a5baf1-58a5-458a-b052-05ec33c00b7f-kube-api-access-w2rjr\") pod \"00a5baf1-58a5-458a-b052-05ec33c00b7f\" (UID: \"00a5baf1-58a5-458a-b052-05ec33c00b7f\") " Sep 11 19:44:30.605281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.605247 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a5baf1-58a5-458a-b052-05ec33c00b7f-kube-api-access-w2rjr" (OuterVolumeSpecName: "kube-api-access-w2rjr") pod "00a5baf1-58a5-458a-b052-05ec33c00b7f" (UID: "00a5baf1-58a5-458a-b052-05ec33c00b7f"). InnerVolumeSpecName "kube-api-access-w2rjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:44:30.613710 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.613679 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a5baf1-58a5-458a-b052-05ec33c00b7f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "00a5baf1-58a5-458a-b052-05ec33c00b7f" (UID: "00a5baf1-58a5-458a-b052-05ec33c00b7f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:44:30.704508 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.704468 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2rjr\" (UniqueName: \"kubernetes.io/projected/00a5baf1-58a5-458a-b052-05ec33c00b7f-kube-api-access-w2rjr\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:44:30.704508 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:30.704504 2559 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/00a5baf1-58a5-458a-b052-05ec33c00b7f-tls-cert\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:44:31.215307 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.215274 2559 generic.go:356] "Generic (PLEG): container finished" podID="00a5baf1-58a5-458a-b052-05ec33c00b7f" containerID="7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6" exitCode=0 Sep 11 19:44:31.215745 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.215324 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" Sep 11 19:44:31.215745 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.215365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" event={"ID":"00a5baf1-58a5-458a-b052-05ec33c00b7f","Type":"ContainerDied","Data":"7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6"} Sep 11 19:44:31.215745 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.215421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6cf998cbb6-qz7rf" event={"ID":"00a5baf1-58a5-458a-b052-05ec33c00b7f","Type":"ContainerDied","Data":"0273affa96f98d257b40e73f03fd01b537d4c48c8cd6ce55c1ec74f25b2c015e"} Sep 11 19:44:31.215745 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.215477 2559 scope.go:117] "RemoveContainer" containerID="7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6" Sep 11 19:44:31.228539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.228518 2559 scope.go:117] "RemoveContainer" containerID="7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6" Sep 11 19:44:31.228893 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:44:31.228865 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6\": container with ID starting with 7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6 not found: ID does not exist" containerID="7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6" Sep 11 19:44:31.229011 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.228900 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6"} err="failed to get container status \"7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6\": rpc error: code = NotFound desc = could not find container \"7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6\": container with ID starting with 7ca48963efaaaacc11c69abf5cf7bf85c1646f21c5ce1b6494fd5223a6b5b1b6 not found: ID does not exist" Sep 11 19:44:31.242284 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.242254 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6cf998cbb6-qz7rf"] Sep 11 19:44:31.247317 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:31.247282 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6cf998cbb6-qz7rf"] Sep 11 19:44:33.224162 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:33.224130 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a5baf1-58a5-458a-b052-05ec33c00b7f" path="/var/lib/kubelet/pods/00a5baf1-58a5-458a-b052-05ec33c00b7f/volumes" Sep 11 19:44:49.190276 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.190240 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7719-5045b6"] Sep 11 19:44:49.190702 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.190641 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a5baf1-58a5-458a-b052-05ec33c00b7f" containerName="authorino" Sep 11 19:44:49.190702 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.190653 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a5baf1-58a5-458a-b052-05ec33c00b7f" containerName="authorino" Sep 11 19:44:49.190778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.190713 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a5baf1-58a5-458a-b052-05ec33c00b7f" containerName="authorino" Sep 11 19:44:49.194090 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.194071 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-5045b6" Sep 11 19:44:49.197134 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.197115 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xq6qr\"" Sep 11 19:44:49.201676 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.201649 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-5045b6"] Sep 11 19:44:49.270510 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.270471 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7n97\" (UniqueName: \"kubernetes.io/projected/a19a0062-bbaa-45db-bc68-8eff535676e9-kube-api-access-n7n97\") pod \"test-curl-7719-5045b6\" (UID: \"a19a0062-bbaa-45db-bc68-8eff535676e9\") " pod="opendatahub/test-curl-7719-5045b6" Sep 11 19:44:49.371218 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.371184 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7n97\" (UniqueName: \"kubernetes.io/projected/a19a0062-bbaa-45db-bc68-8eff535676e9-kube-api-access-n7n97\") pod \"test-curl-7719-5045b6\" (UID: \"a19a0062-bbaa-45db-bc68-8eff535676e9\") " pod="opendatahub/test-curl-7719-5045b6" Sep 11 19:44:49.380302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.380268 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7n97\" (UniqueName: \"kubernetes.io/projected/a19a0062-bbaa-45db-bc68-8eff535676e9-kube-api-access-n7n97\") pod \"test-curl-7719-5045b6\" (UID: \"a19a0062-bbaa-45db-bc68-8eff535676e9\") " pod="opendatahub/test-curl-7719-5045b6" Sep 11 19:44:49.506246 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.506210 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-5045b6" Sep 11 19:44:49.777923 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:49.777898 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-5045b6"] Sep 11 19:44:49.779863 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:44:49.779834 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda19a0062_bbaa_45db_bc68_8eff535676e9.slice/crio-b644a265b9e274f413c4150d69b68d8961e85cfdfa9f017d237a88ba3391591b WatchSource:0}: Error finding container b644a265b9e274f413c4150d69b68d8961e85cfdfa9f017d237a88ba3391591b: Status 404 returned error can't find the container with id b644a265b9e274f413c4150d69b68d8961e85cfdfa9f017d237a88ba3391591b Sep 11 19:44:50.328281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:50.328247 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-5045b6" event={"ID":"a19a0062-bbaa-45db-bc68-8eff535676e9","Type":"ContainerStarted","Data":"b644a265b9e274f413c4150d69b68d8961e85cfdfa9f017d237a88ba3391591b"} Sep 11 19:44:51.382828 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:51.382780 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:44:52.342901 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.342848 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-5045b6" event={"ID":"a19a0062-bbaa-45db-bc68-8eff535676e9","Type":"ContainerStarted","Data":"765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4"} Sep 11 19:44:52.370898 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.370789 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7719-5045b6" podStartSLOduration=1.047476192 podStartE2EDuration="3.370749266s" totalImagesPullingTime="2.323273074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:52.36926008 +0000 UTC m=+1125.641767280" watchObservedRunningTime="2026-09-11 19:44:52.370749266 +0000 UTC m=+1125.643256466" Sep 11 19:44:52.950328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.950292 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7719-5045b6"] Sep 11 19:44:52.958561 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.958532 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7719-5045b6"] Sep 11 19:44:52.978400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.978368 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87"] Sep 11 19:44:52.978819 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.978804 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a19a0062-bbaa-45db-bc68-8eff535676e9" containerName="test-curl-7719-5045b6" Sep 11 19:44:52.978872 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.978821 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a19a0062-bbaa-45db-bc68-8eff535676e9" containerName="test-curl-7719-5045b6" Sep 11 19:44:52.978910 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.978878 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a19a0062-bbaa-45db-bc68-8eff535676e9" containerName="test-curl-7719-5045b6" Sep 11 19:44:52.981949 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.981931 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:52.984813 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.984796 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ec80ad-dockercfg-z5jqn\"" Sep 11 19:44:52.985161 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.985140 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ec80ad\"" Sep 11 19:44:52.993302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.993255 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr"] Sep 11 19:44:52.996870 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.996849 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:52.999761 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:52.999623 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87"] Sep 11 19:44:53.011842 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.011818 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr"] Sep 11 19:44:53.075754 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.075720 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7719-6fbf90"] Sep 11 19:44:53.079075 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.079058 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-6fbf90" Sep 11 19:44:53.088129 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.088099 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-6fbf90"] Sep 11 19:44:53.107452 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.107396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr\" (UID: \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:53.107598 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.107512 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bca2e9a-8235-4622-9ba5-c771b427917a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87\" (UID: \"0bca2e9a-8235-4622-9ba5-c771b427917a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:53.107598 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.107539 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztpbv\" (UniqueName: \"kubernetes.io/projected/0bca2e9a-8235-4622-9ba5-c771b427917a-kube-api-access-ztpbv\") pod \"payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87\" (UID: \"0bca2e9a-8235-4622-9ba5-c771b427917a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:53.107598 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.107557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zjt\" (UniqueName: \"kubernetes.io/projected/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-kube-api-access-s7zjt\") pod \"payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr\" (UID: \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:53.208355 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.208319 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xjxr\" (UniqueName: \"kubernetes.io/projected/38551813-8201-4ee1-8ddc-59bc41f283c8-kube-api-access-5xjxr\") pod \"test-curl-7719-6fbf90\" (UID: \"38551813-8201-4ee1-8ddc-59bc41f283c8\") " pod="opendatahub/test-curl-7719-6fbf90" Sep 11 19:44:53.208551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.208408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr\" (UID: \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:53.208551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.208458 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bca2e9a-8235-4622-9ba5-c771b427917a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87\" (UID: \"0bca2e9a-8235-4622-9ba5-c771b427917a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:53.208551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.208476 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztpbv\" (UniqueName: \"kubernetes.io/projected/0bca2e9a-8235-4622-9ba5-c771b427917a-kube-api-access-ztpbv\") pod \"payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87\" (UID: \"0bca2e9a-8235-4622-9ba5-c771b427917a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:53.208551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.208493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zjt\" (UniqueName: \"kubernetes.io/projected/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-kube-api-access-s7zjt\") pod \"payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr\" (UID: \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:53.209178 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.209159 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bca2e9a-8235-4622-9ba5-c771b427917a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87\" (UID: \"0bca2e9a-8235-4622-9ba5-c771b427917a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:53.209225 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.209167 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr\" (UID: \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:53.217897 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.217867 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zjt\" (UniqueName: \"kubernetes.io/projected/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-kube-api-access-s7zjt\") pod \"payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr\" (UID: \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:53.218016 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.217872 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztpbv\" (UniqueName: \"kubernetes.io/projected/0bca2e9a-8235-4622-9ba5-c771b427917a-kube-api-access-ztpbv\") pod \"payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87\" (UID: \"0bca2e9a-8235-4622-9ba5-c771b427917a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:53.292674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.292593 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:53.309711 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.309685 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:53.310016 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.309991 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xjxr\" (UniqueName: \"kubernetes.io/projected/38551813-8201-4ee1-8ddc-59bc41f283c8-kube-api-access-5xjxr\") pod \"test-curl-7719-6fbf90\" (UID: \"38551813-8201-4ee1-8ddc-59bc41f283c8\") " pod="opendatahub/test-curl-7719-6fbf90" Sep 11 19:44:53.319491 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.319431 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xjxr\" (UniqueName: \"kubernetes.io/projected/38551813-8201-4ee1-8ddc-59bc41f283c8-kube-api-access-5xjxr\") pod \"test-curl-7719-6fbf90\" (UID: \"38551813-8201-4ee1-8ddc-59bc41f283c8\") " pod="opendatahub/test-curl-7719-6fbf90" Sep 11 19:44:53.350858 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:44:53.350397 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7719-5045b6" container="test-curl-7719-5045b6" Sep 11 19:44:53.388875 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.388845 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-6fbf90" Sep 11 19:44:53.578887 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.578848 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87"] Sep 11 19:44:53.580588 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:44:53.580555 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bca2e9a_8235_4622_9ba5_c771b427917a.slice/crio-061d9006f763e8b0d8cb9138894a6ef9d81c876b07954df165a74e8f93ce23ba WatchSource:0}: Error finding container 061d9006f763e8b0d8cb9138894a6ef9d81c876b07954df165a74e8f93ce23ba: Status 404 returned error can't find the container with id 061d9006f763e8b0d8cb9138894a6ef9d81c876b07954df165a74e8f93ce23ba Sep 11 19:44:53.598090 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.598064 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr"] Sep 11 19:44:53.599516 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:44:53.599485 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a31aba_d6ac_451a_9fb0_b34e81f518c2.slice/crio-e2b817c9eacd4a2d70ecfd267af134458e9c7f998040a969f83a378c986537be WatchSource:0}: Error finding container e2b817c9eacd4a2d70ecfd267af134458e9c7f998040a969f83a378c986537be: Status 404 returned error can't find the container with id e2b817c9eacd4a2d70ecfd267af134458e9c7f998040a969f83a378c986537be Sep 11 19:44:53.667884 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:53.667856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7719-6fbf90"] Sep 11 19:44:53.669681 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:44:53.669658 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38551813_8201_4ee1_8ddc_59bc41f283c8.slice/crio-3996ddfb4ce88447e2a5c4cf34a762702653696b404ba38a6ac258c822e39e94 WatchSource:0}: Error finding container 3996ddfb4ce88447e2a5c4cf34a762702653696b404ba38a6ac258c822e39e94: Status 404 returned error can't find the container with id 3996ddfb4ce88447e2a5c4cf34a762702653696b404ba38a6ac258c822e39e94 Sep 11 19:44:54.360237 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:54.360197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-6fbf90" event={"ID":"38551813-8201-4ee1-8ddc-59bc41f283c8","Type":"ContainerStarted","Data":"e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1"} Sep 11 19:44:54.360712 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:54.360250 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7719-6fbf90" event={"ID":"38551813-8201-4ee1-8ddc-59bc41f283c8","Type":"ContainerStarted","Data":"3996ddfb4ce88447e2a5c4cf34a762702653696b404ba38a6ac258c822e39e94"} Sep 11 19:44:54.361964 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:54.361938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" event={"ID":"0bca2e9a-8235-4622-9ba5-c771b427917a","Type":"ContainerStarted","Data":"061d9006f763e8b0d8cb9138894a6ef9d81c876b07954df165a74e8f93ce23ba"} Sep 11 19:44:54.363847 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:54.363472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" event={"ID":"f2a31aba-d6ac-451a-9fb0-b34e81f518c2","Type":"ContainerStarted","Data":"e2b817c9eacd4a2d70ecfd267af134458e9c7f998040a969f83a378c986537be"} Sep 11 19:44:54.363847 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:54.363533 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7719-5045b6" podUID="a19a0062-bbaa-45db-bc68-8eff535676e9" containerName="test-curl-7719-5045b6" containerID="cri-o://765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4" gracePeriod=2 Sep 11 19:44:54.380087 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:54.380041 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7719-6fbf90" podStartSLOduration=1.380024526 podStartE2EDuration="1.380024526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:54.377057322 +0000 UTC m=+1127.649564526" watchObservedRunningTime="2026-09-11 19:44:54.380024526 +0000 UTC m=+1127.652531727" Sep 11 19:44:55.066620 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.066534 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7719-6fbf90"] Sep 11 19:44:55.072954 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.072918 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7719-6fbf90"] Sep 11 19:44:55.103019 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.102971 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h"] Sep 11 19:44:55.103495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.103474 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38551813-8201-4ee1-8ddc-59bc41f283c8" containerName="test-curl-7719-6fbf90" Sep 11 19:44:55.103495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.103496 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="38551813-8201-4ee1-8ddc-59bc41f283c8" containerName="test-curl-7719-6fbf90" Sep 11 19:44:55.103674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.103614 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="38551813-8201-4ee1-8ddc-59bc41f283c8" containerName="test-curl-7719-6fbf90" Sep 11 19:44:55.106884 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.106812 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m"] Sep 11 19:44:55.107545 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.107523 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:55.117561 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.116044 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-c7766c\"" Sep 11 19:44:55.117561 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.116104 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:55.142358 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.142329 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h"] Sep 11 19:44:55.144821 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.144796 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m"] Sep 11 19:44:55.226540 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.226504 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h\" (UID: \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:55.226540 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.226550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl6dg\" (UniqueName: \"kubernetes.io/projected/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-kube-api-access-wl6dg\") pod \"payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h\" (UID: \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:55.226800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.226661 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m\" (UID: \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:55.226800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.226678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ff99\" (UniqueName: \"kubernetes.io/projected/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-kube-api-access-2ff99\") pod \"payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m\" (UID: \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:55.328214 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.328114 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h\" (UID: \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:55.328214 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.328170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl6dg\" (UniqueName: \"kubernetes.io/projected/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-kube-api-access-wl6dg\") pod \"payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h\" (UID: \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:55.328463 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.328236 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m\" (UID: \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:55.328463 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.328328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ff99\" (UniqueName: \"kubernetes.io/projected/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-kube-api-access-2ff99\") pod \"payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m\" (UID: \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:55.328984 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.328964 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h\" (UID: \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:55.329072 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.329053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m\" (UID: \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:55.337007 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.336974 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ff99\" (UniqueName: \"kubernetes.io/projected/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-kube-api-access-2ff99\") pod \"payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m\" (UID: \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:55.337250 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.337222 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl6dg\" (UniqueName: \"kubernetes.io/projected/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-kube-api-access-wl6dg\") pod \"payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h\" (UID: \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:55.371137 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.371101 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" event={"ID":"0bca2e9a-8235-4622-9ba5-c771b427917a","Type":"ContainerStarted","Data":"8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83"} Sep 11 19:44:55.371606 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.371289 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:44:55.373017 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.372986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" event={"ID":"f2a31aba-d6ac-451a-9fb0-b34e81f518c2","Type":"ContainerStarted","Data":"3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6"} Sep 11 19:44:55.373146 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.373044 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:44:55.373198 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:44:55.373188 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7719-6fbf90" container="test-curl-7719-6fbf90" Sep 11 19:44:55.391717 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.391661 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" podStartSLOduration=2.112374282 podStartE2EDuration="3.391643887s" totalImagesPullingTime="1.279269605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:55.39120744 +0000 UTC m=+1128.663714641" watchObservedRunningTime="2026-09-11 19:44:55.391643887 +0000 UTC m=+1128.664151086" Sep 11 19:44:55.409001 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.408938 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" podStartSLOduration=2.146293015 podStartE2EDuration="3.40891448s" totalImagesPullingTime="1.262621465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:55.40885197 +0000 UTC m=+1128.681359171" watchObservedRunningTime="2026-09-11 19:44:55.40891448 +0000 UTC m=+1128.681421679" Sep 11 19:44:55.426032 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.425961 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:55.432204 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.432175 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:55.711634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.711602 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h"] Sep 11 19:44:55.712168 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:44:55.712140 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ec1ef0_b03b_4a04_abf1_3fa38c6de0cd.slice/crio-b5083ff0567e8106fb8cc319d25c249d14d70b54cef8c48c8e75db95c1a32af7 WatchSource:0}: Error finding container b5083ff0567e8106fb8cc319d25c249d14d70b54cef8c48c8e75db95c1a32af7: Status 404 returned error can't find the container with id b5083ff0567e8106fb8cc319d25c249d14d70b54cef8c48c8e75db95c1a32af7 Sep 11 19:44:55.732576 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:55.732547 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m"] Sep 11 19:44:55.733557 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:44:55.733528 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b5285cd_e61b_4620_ac4f_b60e0372ef6c.slice/crio-fa91e2c8b8150eb15b1c9aaf9762a127ffe48012c371060de46c498a941ae5c6 WatchSource:0}: Error finding container fa91e2c8b8150eb15b1c9aaf9762a127ffe48012c371060de46c498a941ae5c6: Status 404 returned error can't find the container with id fa91e2c8b8150eb15b1c9aaf9762a127ffe48012c371060de46c498a941ae5c6 Sep 11 19:44:56.382274 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.382249 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" event={"ID":"8b5285cd-e61b-4620-ac4f-b60e0372ef6c","Type":"ContainerStarted","Data":"6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197"} Sep 11 19:44:56.382598 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.382296 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" event={"ID":"8b5285cd-e61b-4620-ac4f-b60e0372ef6c","Type":"ContainerStarted","Data":"fa91e2c8b8150eb15b1c9aaf9762a127ffe48012c371060de46c498a941ae5c6"} Sep 11 19:44:56.382598 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.382459 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:44:56.384084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.384056 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" event={"ID":"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd","Type":"ContainerStarted","Data":"80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be"} Sep 11 19:44:56.384175 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.384085 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7719-6fbf90" podUID="38551813-8201-4ee1-8ddc-59bc41f283c8" containerName="test-curl-7719-6fbf90" containerID="cri-o://e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1" gracePeriod=2 Sep 11 19:44:56.384357 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.384337 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:44:56.384496 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.384376 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" event={"ID":"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd","Type":"ContainerStarted","Data":"b5083ff0567e8106fb8cc319d25c249d14d70b54cef8c48c8e75db95c1a32af7"} Sep 11 19:44:56.406971 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.406093 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" podStartSLOduration=1.406071098 podStartE2EDuration="1.406071098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:56.405738949 +0000 UTC m=+1129.678246151" watchObservedRunningTime="2026-09-11 19:44:56.406071098 +0000 UTC m=+1129.678578300" Sep 11 19:44:56.428725 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.428654 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" podStartSLOduration=1.428632125 podStartE2EDuration="1.428632125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:44:56.426278286 +0000 UTC m=+1129.698785512" watchObservedRunningTime="2026-09-11 19:44:56.428632125 +0000 UTC m=+1129.701139327" Sep 11 19:44:56.546817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.546783 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-5045b6" Sep 11 19:44:56.641150 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.641076 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7n97\" (UniqueName: \"kubernetes.io/projected/a19a0062-bbaa-45db-bc68-8eff535676e9-kube-api-access-n7n97\") pod \"a19a0062-bbaa-45db-bc68-8eff535676e9\" (UID: \"a19a0062-bbaa-45db-bc68-8eff535676e9\") " Sep 11 19:44:56.643339 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.643317 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a19a0062-bbaa-45db-bc68-8eff535676e9-kube-api-access-n7n97" (OuterVolumeSpecName: "kube-api-access-n7n97") pod "a19a0062-bbaa-45db-bc68-8eff535676e9" (UID: "a19a0062-bbaa-45db-bc68-8eff535676e9"). InnerVolumeSpecName "kube-api-access-n7n97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:44:56.742640 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:56.742605 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7n97\" (UniqueName: \"kubernetes.io/projected/a19a0062-bbaa-45db-bc68-8eff535676e9-kube-api-access-n7n97\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:44:57.224702 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:57.224670 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a19a0062-bbaa-45db-bc68-8eff535676e9" path="/var/lib/kubelet/pods/a19a0062-bbaa-45db-bc68-8eff535676e9/volumes" Sep 11 19:44:57.391349 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:57.391315 2559 generic.go:356] "Generic (PLEG): container finished" podID="a19a0062-bbaa-45db-bc68-8eff535676e9" containerID="765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4" exitCode=137 Sep 11 19:44:57.391808 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:57.391362 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-5045b6" Sep 11 19:44:57.391808 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:57.391415 2559 scope.go:117] "RemoveContainer" containerID="765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4" Sep 11 19:44:57.401130 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:57.401109 2559 scope.go:117] "RemoveContainer" containerID="765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4" Sep 11 19:44:57.401476 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:44:57.401451 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4\": container with ID starting with 765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4 not found: ID does not exist" containerID="765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4" Sep 11 19:44:57.401611 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:57.401486 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4"} err="failed to get container status \"765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4\": rpc error: code = NotFound desc = could not find container \"765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4\": container with ID starting with 765a1d08f605ebbbd553b6fbaa935b73a393b15b98deb0515a9c10d4437055d4 not found: ID does not exist" Sep 11 19:44:58.555241 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:58.555212 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-6fbf90" Sep 11 19:44:58.660778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:58.660691 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xjxr\" (UniqueName: \"kubernetes.io/projected/38551813-8201-4ee1-8ddc-59bc41f283c8-kube-api-access-5xjxr\") pod \"38551813-8201-4ee1-8ddc-59bc41f283c8\" (UID: \"38551813-8201-4ee1-8ddc-59bc41f283c8\") " Sep 11 19:44:58.662934 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:58.662907 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38551813-8201-4ee1-8ddc-59bc41f283c8-kube-api-access-5xjxr" (OuterVolumeSpecName: "kube-api-access-5xjxr") pod "38551813-8201-4ee1-8ddc-59bc41f283c8" (UID: "38551813-8201-4ee1-8ddc-59bc41f283c8"). InnerVolumeSpecName "kube-api-access-5xjxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:44:58.761970 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:58.761931 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xjxr\" (UniqueName: \"kubernetes.io/projected/38551813-8201-4ee1-8ddc-59bc41f283c8-kube-api-access-5xjxr\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:44:59.225209 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:59.225174 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38551813-8201-4ee1-8ddc-59bc41f283c8" path="/var/lib/kubelet/pods/38551813-8201-4ee1-8ddc-59bc41f283c8/volumes" Sep 11 19:44:59.406953 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:59.406913 2559 generic.go:356] "Generic (PLEG): container finished" podID="38551813-8201-4ee1-8ddc-59bc41f283c8" containerID="e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1" exitCode=137 Sep 11 19:44:59.407141 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:59.406967 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7719-6fbf90" Sep 11 19:44:59.407141 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:59.406989 2559 scope.go:117] "RemoveContainer" containerID="e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1" Sep 11 19:44:59.418489 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:59.418464 2559 scope.go:117] "RemoveContainer" containerID="e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1" Sep 11 19:44:59.418836 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:44:59.418816 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1\": container with ID starting with e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1 not found: ID does not exist" containerID="e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1" Sep 11 19:44:59.418913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:44:59.418846 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1"} err="failed to get container status \"e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1\": rpc error: code = NotFound desc = could not find container \"e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1\": container with ID starting with e07df801aa7c1af71dec3100967ef721f1602ec0b5b8c13769c018441842e8b1 not found: ID does not exist" Sep 11 19:45:00.155632 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.155594 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l"] Sep 11 19:45:00.161093 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.161065 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w"] Sep 11 19:45:00.161251 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.161187 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" Sep 11 19:45:00.164986 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.164964 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7"] Sep 11 19:45:00.165120 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.165092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" Sep 11 19:45:00.165302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.165285 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:45:00.168679 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.168659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" Sep 11 19:45:00.174852 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.174828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" Sep 11 19:45:00.185334 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.183145 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l"] Sep 11 19:45:00.186092 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.185928 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" Sep 11 19:45:00.188229 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.187272 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w"] Sep 11 19:45:00.192131 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.192062 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7"] Sep 11 19:45:00.239956 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.239660 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" Sep 11 19:45:00.497599 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.497575 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l"] Sep 11 19:45:00.499961 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:00.499930 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd175fbbd_1399_477b_b320_8abb5cc53ec9.slice/crio-4c7da8b3aacc63333cf4703d5a94322796134cbf59f37e45fd02ef71e874ee6b WatchSource:0}: Error finding container 4c7da8b3aacc63333cf4703d5a94322796134cbf59f37e45fd02ef71e874ee6b: Status 404 returned error can't find the container with id 4c7da8b3aacc63333cf4703d5a94322796134cbf59f37e45fd02ef71e874ee6b Sep 11 19:45:00.513835 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.513812 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w"] Sep 11 19:45:00.515948 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:00.515907 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc858de9_7553_403d_82dc_9d9a16d971f7.slice/crio-b598bf78fda4db735ca31455a9d612dffd7464b9874e92d0d0dcb3a7a5bbe088 WatchSource:0}: Error finding container b598bf78fda4db735ca31455a9d612dffd7464b9874e92d0d0dcb3a7a5bbe088: Status 404 returned error can't find the container with id b598bf78fda4db735ca31455a9d612dffd7464b9874e92d0d0dcb3a7a5bbe088 Sep 11 19:45:00.568790 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:00.568755 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb702102_1ea1_4244_9b8e_149f1a94f2f6.slice/crio-303c3bf8cf73277fb6bf4725de02bad3dd475c3de2002b8639f58389d8f041cd WatchSource:0}: Error finding container 303c3bf8cf73277fb6bf4725de02bad3dd475c3de2002b8639f58389d8f041cd: Status 404 returned error can't find the container with id 303c3bf8cf73277fb6bf4725de02bad3dd475c3de2002b8639f58389d8f041cd Sep 11 19:45:00.583316 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:00.583284 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7"] Sep 11 19:45:01.433294 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:01.433205 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" event={"ID":"cc858de9-7553-403d-82dc-9d9a16d971f7","Type":"ContainerStarted","Data":"b598bf78fda4db735ca31455a9d612dffd7464b9874e92d0d0dcb3a7a5bbe088"} Sep 11 19:45:01.435859 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:01.435811 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" event={"ID":"db702102-1ea1-4244-9b8e-149f1a94f2f6","Type":"ContainerStarted","Data":"303c3bf8cf73277fb6bf4725de02bad3dd475c3de2002b8639f58389d8f041cd"} Sep 11 19:45:01.438140 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:01.438061 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" event={"ID":"d175fbbd-1399-477b-b320-8abb5cc53ec9","Type":"ContainerStarted","Data":"4c7da8b3aacc63333cf4703d5a94322796134cbf59f37e45fd02ef71e874ee6b"} Sep 11 19:45:03.455064 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:03.455032 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" event={"ID":"cc858de9-7553-403d-82dc-9d9a16d971f7","Type":"ContainerStarted","Data":"f7a7c58a4dafd77569d186e619c127cb61159f5375c3599a24cb5ced7845bb69"} Sep 11 19:45:03.456814 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:03.456785 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" event={"ID":"db702102-1ea1-4244-9b8e-149f1a94f2f6","Type":"ContainerStarted","Data":"c94b4f2935f4437800346292caaf74eef92b98497bb431107154ede841fe8929"} Sep 11 19:45:03.458360 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:03.458331 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" event={"ID":"d175fbbd-1399-477b-b320-8abb5cc53ec9","Type":"ContainerStarted","Data":"0e18ae8d10d81c9a396b981a78ac6564ef6b624302e350c04d427946ab86c699"} Sep 11 19:45:03.477293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:03.477215 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" podStartSLOduration=1.042166699 podStartE2EDuration="3.477201763s" totalImagesPullingTime="2.435035064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:03.47628068 +0000 UTC m=+1136.748787879" watchObservedRunningTime="2026-09-11 19:45:03.477201763 +0000 UTC m=+1136.749708962" Sep 11 19:45:03.524862 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:03.524796 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" podStartSLOduration=1.074762678 podStartE2EDuration="3.524778395s" totalImagesPullingTime="2.450015717s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:03.522522108 +0000 UTC m=+1136.795029307" watchObservedRunningTime="2026-09-11 19:45:03.524778395 +0000 UTC m=+1136.797285611" Sep 11 19:45:03.563598 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:03.563540 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" podStartSLOduration=1.181198263 podStartE2EDuration="3.563525639s" totalImagesPullingTime="2.382327376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:03.561025831 +0000 UTC m=+1136.833533041" watchObservedRunningTime="2026-09-11 19:45:03.563525639 +0000 UTC m=+1136.836032883" Sep 11 19:45:04.466257 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:04.466173 2559 generic.go:356] "Generic (PLEG): container finished" podID="db702102-1ea1-4244-9b8e-149f1a94f2f6" containerID="c94b4f2935f4437800346292caaf74eef92b98497bb431107154ede841fe8929" exitCode=0 Sep 11 19:45:04.466660 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:04.466256 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" event={"ID":"db702102-1ea1-4244-9b8e-149f1a94f2f6","Type":"ContainerDied","Data":"c94b4f2935f4437800346292caaf74eef92b98497bb431107154ede841fe8929"} Sep 11 19:45:04.467913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:04.467886 2559 generic.go:356] "Generic (PLEG): container finished" podID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerID="0e18ae8d10d81c9a396b981a78ac6564ef6b624302e350c04d427946ab86c699" exitCode=7 Sep 11 19:45:04.468045 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:04.467970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" event={"ID":"d175fbbd-1399-477b-b320-8abb5cc53ec9","Type":"ContainerDied","Data":"0e18ae8d10d81c9a396b981a78ac6564ef6b624302e350c04d427946ab86c699"} Sep 11 19:45:04.468129 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:04.468113 2559 scope.go:117] "RemoveContainer" containerID="0e18ae8d10d81c9a396b981a78ac6564ef6b624302e350c04d427946ab86c699" Sep 11 19:45:04.469683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:04.469659 2559 generic.go:356] "Generic (PLEG): container finished" podID="cc858de9-7553-403d-82dc-9d9a16d971f7" containerID="f7a7c58a4dafd77569d186e619c127cb61159f5375c3599a24cb5ced7845bb69" exitCode=0 Sep 11 19:45:04.469764 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:04.469691 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" event={"ID":"cc858de9-7553-403d-82dc-9d9a16d971f7","Type":"ContainerDied","Data":"f7a7c58a4dafd77569d186e619c127cb61159f5375c3599a24cb5ced7845bb69"} Sep 11 19:45:05.477895 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:05.477862 2559 generic.go:356] "Generic (PLEG): container finished" podID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerID="69dadf37c199cc6f33d620e67fd426e53234ed8f1413f6c56bd6a8df74ab8cae" exitCode=7 Sep 11 19:45:05.478308 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:05.477920 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" event={"ID":"d175fbbd-1399-477b-b320-8abb5cc53ec9","Type":"ContainerDied","Data":"69dadf37c199cc6f33d620e67fd426e53234ed8f1413f6c56bd6a8df74ab8cae"} Sep 11 19:45:05.478308 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:05.477977 2559 scope.go:117] "RemoveContainer" containerID="0e18ae8d10d81c9a396b981a78ac6564ef6b624302e350c04d427946ab86c699" Sep 11 19:45:05.478308 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:05.478116 2559 scope.go:117] "RemoveContainer" containerID="69dadf37c199cc6f33d620e67fd426e53234ed8f1413f6c56bd6a8df74ab8cae" Sep 11 19:45:05.478494 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:45:05.478415 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l_odh-ai-gateway-infra(d175fbbd-1399-477b-b320-8abb5cc53ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" Sep 11 19:45:05.640338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:05.640316 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" Sep 11 19:45:05.642616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:05.642590 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" Sep 11 19:45:06.333131 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.333095 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk"] Sep 11 19:45:06.333538 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.333525 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db702102-1ea1-4244-9b8e-149f1a94f2f6" containerName="cleanup" Sep 11 19:45:06.333538 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.333539 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="db702102-1ea1-4244-9b8e-149f1a94f2f6" containerName="cleanup" Sep 11 19:45:06.333639 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.333550 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc858de9-7553-403d-82dc-9d9a16d971f7" containerName="cleanup" Sep 11 19:45:06.333639 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.333556 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc858de9-7553-403d-82dc-9d9a16d971f7" containerName="cleanup" Sep 11 19:45:06.333639 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.333614 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="db702102-1ea1-4244-9b8e-149f1a94f2f6" containerName="cleanup" Sep 11 19:45:06.333639 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.333624 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc858de9-7553-403d-82dc-9d9a16d971f7" containerName="cleanup" Sep 11 19:45:06.336823 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.336803 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:06.339928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.339911 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-6bf598-dockercfg-c4wxk\"" Sep 11 19:45:06.340287 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.340272 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-6bf598\"" Sep 11 19:45:06.356722 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.356699 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk"] Sep 11 19:45:06.387004 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.386963 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:45:06.387453 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.387418 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:45:06.387617 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.387598 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj"] Sep 11 19:45:06.391044 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.391026 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:06.406482 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.406429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj"] Sep 11 19:45:06.437102 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.437063 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk\" (UID: \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:06.437288 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.437118 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8jj\" (UniqueName: \"kubernetes.io/projected/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-kube-api-access-cv8jj\") pod \"payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk\" (UID: \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:06.494552 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.494519 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" Sep 11 19:45:06.494552 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.494532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29819265-hpn2w" event={"ID":"cc858de9-7553-403d-82dc-9d9a16d971f7","Type":"ContainerDied","Data":"b598bf78fda4db735ca31455a9d612dffd7464b9874e92d0d0dcb3a7a5bbe088"} Sep 11 19:45:06.495034 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.494620 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b598bf78fda4db735ca31455a9d612dffd7464b9874e92d0d0dcb3a7a5bbe088" Sep 11 19:45:06.496216 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.496196 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" Sep 11 19:45:06.496216 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.496202 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7" event={"ID":"db702102-1ea1-4244-9b8e-149f1a94f2f6","Type":"ContainerDied","Data":"303c3bf8cf73277fb6bf4725de02bad3dd475c3de2002b8639f58389d8f041cd"} Sep 11 19:45:06.496407 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.496242 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="303c3bf8cf73277fb6bf4725de02bad3dd475c3de2002b8639f58389d8f041cd" Sep 11 19:45:06.498267 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.498248 2559 scope.go:117] "RemoveContainer" containerID="69dadf37c199cc6f33d620e67fd426e53234ed8f1413f6c56bd6a8df74ab8cae" Sep 11 19:45:06.498503 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:45:06.498479 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l_odh-ai-gateway-infra(d175fbbd-1399-477b-b320-8abb5cc53ec9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" Sep 11 19:45:06.537800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.537762 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk8vx\" (UniqueName: \"kubernetes.io/projected/45ca6c7f-0688-4d27-a70c-77df94606890-kube-api-access-fk8vx\") pod \"payload-processing-e2e-autores-6bf598-855464d6f7-88jmj\" (UID: \"45ca6c7f-0688-4d27-a70c-77df94606890\") " pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:06.538005 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.537829 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca6c7f-0688-4d27-a70c-77df94606890-plugins-config-volume\") pod \"payload-processing-e2e-autores-6bf598-855464d6f7-88jmj\" (UID: \"45ca6c7f-0688-4d27-a70c-77df94606890\") " pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:06.538005 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.537895 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk\" (UID: \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:06.538005 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.537934 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8jj\" (UniqueName: \"kubernetes.io/projected/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-kube-api-access-cv8jj\") pod \"payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk\" (UID: \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:06.538763 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.538731 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk\" (UID: \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:06.546979 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.546949 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8jj\" (UniqueName: \"kubernetes.io/projected/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-kube-api-access-cv8jj\") pod \"payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk\" (UID: \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:06.638841 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.638751 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca6c7f-0688-4d27-a70c-77df94606890-plugins-config-volume\") pod \"payload-processing-e2e-autores-6bf598-855464d6f7-88jmj\" (UID: \"45ca6c7f-0688-4d27-a70c-77df94606890\") " pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:06.638998 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.638872 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk8vx\" (UniqueName: \"kubernetes.io/projected/45ca6c7f-0688-4d27-a70c-77df94606890-kube-api-access-fk8vx\") pod \"payload-processing-e2e-autores-6bf598-855464d6f7-88jmj\" (UID: \"45ca6c7f-0688-4d27-a70c-77df94606890\") " pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:06.639396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.639374 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca6c7f-0688-4d27-a70c-77df94606890-plugins-config-volume\") pod \"payload-processing-e2e-autores-6bf598-855464d6f7-88jmj\" (UID: \"45ca6c7f-0688-4d27-a70c-77df94606890\") " pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:06.647698 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.647682 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:06.650560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.650536 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk8vx\" (UniqueName: \"kubernetes.io/projected/45ca6c7f-0688-4d27-a70c-77df94606890-kube-api-access-fk8vx\") pod \"payload-processing-e2e-autores-6bf598-855464d6f7-88jmj\" (UID: \"45ca6c7f-0688-4d27-a70c-77df94606890\") " pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:06.701989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.701948 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:06.925870 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.925841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk"] Sep 11 19:45:06.927293 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:06.927269 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67f04ff2_d7ee_45a1_a3d3_f435213a4a89.slice/crio-6e4b99b7d221200465f3d2ff07b8d89b899575d82591723253521befc17e1f23 WatchSource:0}: Error finding container 6e4b99b7d221200465f3d2ff07b8d89b899575d82591723253521befc17e1f23: Status 404 returned error can't find the container with id 6e4b99b7d221200465f3d2ff07b8d89b899575d82591723253521befc17e1f23 Sep 11 19:45:06.990359 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:06.990332 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj"] Sep 11 19:45:06.991315 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:06.991286 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ca6c7f_0688_4d27_a70c_77df94606890.slice/crio-b7cfc0b905f53bf971c10801b9171dc4fa1ce9858032506600a37596b3fa046e WatchSource:0}: Error finding container b7cfc0b905f53bf971c10801b9171dc4fa1ce9858032506600a37596b3fa046e: Status 404 returned error can't find the container with id b7cfc0b905f53bf971c10801b9171dc4fa1ce9858032506600a37596b3fa046e Sep 11 19:45:07.393390 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.393359 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:45:07.393559 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.393454 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:45:07.507687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.507644 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" event={"ID":"45ca6c7f-0688-4d27-a70c-77df94606890","Type":"ContainerStarted","Data":"14b655da7bdf24e53d04f548f577d9ab929773fc85908b45f9096b21d128721a"} Sep 11 19:45:07.508122 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.507749 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" event={"ID":"45ca6c7f-0688-4d27-a70c-77df94606890","Type":"ContainerStarted","Data":"b7cfc0b905f53bf971c10801b9171dc4fa1ce9858032506600a37596b3fa046e"} Sep 11 19:45:07.508122 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.507802 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:07.510271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.510241 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" event={"ID":"67f04ff2-d7ee-45a1-a3d3-f435213a4a89","Type":"ContainerStarted","Data":"8562547ae49616014117cbd0640a27ea1ce7571d565bc8d9ced634a1fb144bce"} Sep 11 19:45:07.510467 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.510319 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:07.510467 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.510346 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" event={"ID":"67f04ff2-d7ee-45a1-a3d3-f435213a4a89","Type":"ContainerStarted","Data":"6e4b99b7d221200465f3d2ff07b8d89b899575d82591723253521befc17e1f23"} Sep 11 19:45:07.529577 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.529510 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" podStartSLOduration=1.5294883019999999 podStartE2EDuration="1.529488302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:07.527188035 +0000 UTC m=+1140.799695238" watchObservedRunningTime="2026-09-11 19:45:07.529488302 +0000 UTC m=+1140.801995503" Sep 11 19:45:07.558894 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.558781 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" podStartSLOduration=1.558758791 podStartE2EDuration="1.558758791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:07.553700455 +0000 UTC m=+1140.826207656" watchObservedRunningTime="2026-09-11 19:45:07.558758791 +0000 UTC m=+1140.831265992" Sep 11 19:45:07.744734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.744690 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz"] Sep 11 19:45:07.749765 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.749748 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:07.752774 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.752748 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4c47b5e2-dockercfg-lj2ws\"" Sep 11 19:45:07.754350 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.754325 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4c47b5e2\"" Sep 11 19:45:07.778940 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.778915 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz"] Sep 11 19:45:07.853341 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.853251 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6mw\" (UniqueName: \"kubernetes.io/projected/86938d9a-1cdb-456a-9318-ba427f01b68e-kube-api-access-gm6mw\") pod \"payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz\" (UID: \"86938d9a-1cdb-456a-9318-ba427f01b68e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:07.853341 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.853315 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86938d9a-1cdb-456a-9318-ba427f01b68e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz\" (UID: \"86938d9a-1cdb-456a-9318-ba427f01b68e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:07.954616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.954575 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86938d9a-1cdb-456a-9318-ba427f01b68e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz\" (UID: \"86938d9a-1cdb-456a-9318-ba427f01b68e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:07.954767 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.954670 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6mw\" (UniqueName: \"kubernetes.io/projected/86938d9a-1cdb-456a-9318-ba427f01b68e-kube-api-access-gm6mw\") pod \"payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz\" (UID: \"86938d9a-1cdb-456a-9318-ba427f01b68e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:07.955576 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.955546 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86938d9a-1cdb-456a-9318-ba427f01b68e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz\" (UID: \"86938d9a-1cdb-456a-9318-ba427f01b68e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:07.963690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:07.963660 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6mw\" (UniqueName: \"kubernetes.io/projected/86938d9a-1cdb-456a-9318-ba427f01b68e-kube-api-access-gm6mw\") pod \"payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz\" (UID: \"86938d9a-1cdb-456a-9318-ba427f01b68e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:08.061220 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:08.061183 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:08.360624 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:08.360589 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz"] Sep 11 19:45:08.361598 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:08.361571 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86938d9a_1cdb_456a_9318_ba427f01b68e.slice/crio-ca123b20dd3b2ca4c16800e6224eaa7ce317fefbb78866738efb71bb4e7cced6 WatchSource:0}: Error finding container ca123b20dd3b2ca4c16800e6224eaa7ce317fefbb78866738efb71bb4e7cced6: Status 404 returned error can't find the container with id ca123b20dd3b2ca4c16800e6224eaa7ce317fefbb78866738efb71bb4e7cced6 Sep 11 19:45:08.518671 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:08.518630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" event={"ID":"86938d9a-1cdb-456a-9318-ba427f01b68e","Type":"ContainerStarted","Data":"02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562"} Sep 11 19:45:08.519153 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:08.518700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" event={"ID":"86938d9a-1cdb-456a-9318-ba427f01b68e","Type":"ContainerStarted","Data":"ca123b20dd3b2ca4c16800e6224eaa7ce317fefbb78866738efb71bb4e7cced6"} Sep 11 19:45:08.519153 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:08.518963 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:08.542739 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:08.542680 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" podStartSLOduration=1.542661936 podStartE2EDuration="1.542661936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:08.541103655 +0000 UTC m=+1141.813610858" watchObservedRunningTime="2026-09-11 19:45:08.542661936 +0000 UTC m=+1141.815169176" Sep 11 19:45:11.372586 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:11.372547 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:45:16.334018 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.333976 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf"] Sep 11 19:45:16.341192 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.341161 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" Sep 11 19:45:16.344890 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.344858 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:45:16.347358 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.347334 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf"] Sep 11 19:45:16.437132 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.437093 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec643558-a83e-4960-b646-ace228c51be0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf\" (UID: \"ec643558-a83e-4960-b646-ace228c51be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" Sep 11 19:45:16.538323 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.538286 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec643558-a83e-4960-b646-ace228c51be0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf\" (UID: \"ec643558-a83e-4960-b646-ace228c51be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" Sep 11 19:45:16.538997 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.538975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec643558-a83e-4960-b646-ace228c51be0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf\" (UID: \"ec643558-a83e-4960-b646-ace228c51be0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" Sep 11 19:45:16.586895 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.586798 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw"] Sep 11 19:45:16.591366 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.591340 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:16.594353 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.594329 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-872425\"" Sep 11 19:45:16.594533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.594329 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-872425-dockercfg-7wdv8\"" Sep 11 19:45:16.604384 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.604357 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw"] Sep 11 19:45:16.653936 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.653905 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" Sep 11 19:45:16.740147 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.740112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cb86bd-84ad-4a2f-a884-6f43c41b75de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw\" (UID: \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:16.740331 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.740150 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gkcr\" (UniqueName: \"kubernetes.io/projected/32cb86bd-84ad-4a2f-a884-6f43c41b75de-kube-api-access-2gkcr\") pod \"payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw\" (UID: \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:16.799935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.799901 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27"] Sep 11 19:45:16.805393 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.805375 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:16.816043 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.816014 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27"] Sep 11 19:45:16.841534 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.841445 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cb86bd-84ad-4a2f-a884-6f43c41b75de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw\" (UID: \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:16.841534 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.841481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gkcr\" (UniqueName: \"kubernetes.io/projected/32cb86bd-84ad-4a2f-a884-6f43c41b75de-kube-api-access-2gkcr\") pod \"payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw\" (UID: \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:16.842416 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.842394 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cb86bd-84ad-4a2f-a884-6f43c41b75de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw\" (UID: \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:16.851710 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.851687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gkcr\" (UniqueName: \"kubernetes.io/projected/32cb86bd-84ad-4a2f-a884-6f43c41b75de-kube-api-access-2gkcr\") pod \"payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw\" (UID: \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:16.904293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.904263 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:16.942235 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.942201 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4435c463-fcd7-494d-9d19-34270da006f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27\" (UID: \"4435c463-fcd7-494d-9d19-34270da006f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:16.942410 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.942259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfpmf\" (UniqueName: \"kubernetes.io/projected/4435c463-fcd7-494d-9d19-34270da006f1-kube-api-access-jfpmf\") pod \"payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27\" (UID: \"4435c463-fcd7-494d-9d19-34270da006f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:16.955678 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:16.955644 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf"] Sep 11 19:45:17.043034 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.042996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfpmf\" (UniqueName: \"kubernetes.io/projected/4435c463-fcd7-494d-9d19-34270da006f1-kube-api-access-jfpmf\") pod \"payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27\" (UID: \"4435c463-fcd7-494d-9d19-34270da006f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:17.043229 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.043099 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4435c463-fcd7-494d-9d19-34270da006f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27\" (UID: \"4435c463-fcd7-494d-9d19-34270da006f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:17.043785 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.043765 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4435c463-fcd7-494d-9d19-34270da006f1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27\" (UID: \"4435c463-fcd7-494d-9d19-34270da006f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:17.056661 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.056635 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfpmf\" (UniqueName: \"kubernetes.io/projected/4435c463-fcd7-494d-9d19-34270da006f1-kube-api-access-jfpmf\") pod \"payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27\" (UID: \"4435c463-fcd7-494d-9d19-34270da006f1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:17.118742 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.118717 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:17.202573 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.202542 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw"] Sep 11 19:45:17.207981 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:17.207952 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cb86bd_84ad_4a2f_a884_6f43c41b75de.slice/crio-79eb34a68e39773228604a78290e959f3bb41583eb88f83b3d190f58fb60dcc9 WatchSource:0}: Error finding container 79eb34a68e39773228604a78290e959f3bb41583eb88f83b3d190f58fb60dcc9: Status 404 returned error can't find the container with id 79eb34a68e39773228604a78290e959f3bb41583eb88f83b3d190f58fb60dcc9 Sep 11 19:45:17.397604 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.397576 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27"] Sep 11 19:45:17.398669 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:17.398616 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4435c463_fcd7_494d_9d19_34270da006f1.slice/crio-6a5ed285c9e2b4eec956ce4d1a1229a84449a91246aa3ff38578e4adcdbac0a5 WatchSource:0}: Error finding container 6a5ed285c9e2b4eec956ce4d1a1229a84449a91246aa3ff38578e4adcdbac0a5: Status 404 returned error can't find the container with id 6a5ed285c9e2b4eec956ce4d1a1229a84449a91246aa3ff38578e4adcdbac0a5 Sep 11 19:45:17.578362 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.578322 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" event={"ID":"4435c463-fcd7-494d-9d19-34270da006f1","Type":"ContainerStarted","Data":"423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9"} Sep 11 19:45:17.578574 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.578426 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:17.578574 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.578469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" event={"ID":"4435c463-fcd7-494d-9d19-34270da006f1","Type":"ContainerStarted","Data":"6a5ed285c9e2b4eec956ce4d1a1229a84449a91246aa3ff38578e4adcdbac0a5"} Sep 11 19:45:17.580226 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.580198 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" event={"ID":"32cb86bd-84ad-4a2f-a884-6f43c41b75de","Type":"ContainerStarted","Data":"c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a"} Sep 11 19:45:17.580379 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.580252 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:17.580379 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.580275 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" event={"ID":"32cb86bd-84ad-4a2f-a884-6f43c41b75de","Type":"ContainerStarted","Data":"79eb34a68e39773228604a78290e959f3bb41583eb88f83b3d190f58fb60dcc9"} Sep 11 19:45:17.581900 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.581872 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" event={"ID":"ec643558-a83e-4960-b646-ace228c51be0","Type":"ContainerStarted","Data":"f1403d0de3a6efdcf6d81ad863af3da3e47f47f788aa49ef3b55ef7699c48471"} Sep 11 19:45:17.581991 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.581928 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" event={"ID":"ec643558-a83e-4960-b646-ace228c51be0","Type":"ContainerStarted","Data":"c09d12ebe8ac6488cd6dad0b2f4cf6a3b5840a0c05349bf73b37cef93b954f88"} Sep 11 19:45:17.617154 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.617106 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" podStartSLOduration=1.617089671 podStartE2EDuration="1.617089671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:17.61537402 +0000 UTC m=+1150.887881246" watchObservedRunningTime="2026-09-11 19:45:17.617089671 +0000 UTC m=+1150.889596870" Sep 11 19:45:17.662830 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.662718 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" podStartSLOduration=1.662697823 podStartE2EDuration="1.662697823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:17.657630797 +0000 UTC m=+1150.930137997" watchObservedRunningTime="2026-09-11 19:45:17.662697823 +0000 UTC m=+1150.935205027" Sep 11 19:45:17.751575 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:17.751523 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" podStartSLOduration=1.751507303 podStartE2EDuration="1.751507303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:17.71829126 +0000 UTC m=+1150.990798459" watchObservedRunningTime="2026-09-11 19:45:17.751507303 +0000 UTC m=+1151.024014503" Sep 11 19:45:18.521575 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:18.521544 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:45:18.522014 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:18.521614 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:45:18.591926 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:18.591894 2559 generic.go:356] "Generic (PLEG): container finished" podID="ec643558-a83e-4960-b646-ace228c51be0" containerID="f1403d0de3a6efdcf6d81ad863af3da3e47f47f788aa49ef3b55ef7699c48471" exitCode=0 Sep 11 19:45:18.592094 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:18.591967 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" event={"ID":"ec643558-a83e-4960-b646-ace228c51be0","Type":"ContainerDied","Data":"f1403d0de3a6efdcf6d81ad863af3da3e47f47f788aa49ef3b55ef7699c48471"} Sep 11 19:45:19.527954 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:19.527923 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:19.747934 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:19.747911 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" Sep 11 19:45:19.768285 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:19.768258 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec643558-a83e-4960-b646-ace228c51be0-maas-api-service-ca\") pod \"ec643558-a83e-4960-b646-ace228c51be0\" (UID: \"ec643558-a83e-4960-b646-ace228c51be0\") " Sep 11 19:45:19.768665 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:19.768644 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec643558-a83e-4960-b646-ace228c51be0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec643558-a83e-4960-b646-ace228c51be0" (UID: "ec643558-a83e-4960-b646-ace228c51be0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:45:19.869663 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:19.869571 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec643558-a83e-4960-b646-ace228c51be0-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:45:20.607527 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:20.607494 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" Sep 11 19:45:20.607981 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:20.607504 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf" event={"ID":"ec643558-a83e-4960-b646-ace228c51be0","Type":"ContainerDied","Data":"c09d12ebe8ac6488cd6dad0b2f4cf6a3b5840a0c05349bf73b37cef93b954f88"} Sep 11 19:45:20.607981 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:20.607616 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09d12ebe8ac6488cd6dad0b2f4cf6a3b5840a0c05349bf73b37cef93b954f88" Sep 11 19:45:22.217940 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:22.217902 2559 scope.go:117] "RemoveContainer" containerID="69dadf37c199cc6f33d620e67fd426e53234ed8f1413f6c56bd6a8df74ab8cae" Sep 11 19:45:22.622093 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:22.622064 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" event={"ID":"d175fbbd-1399-477b-b320-8abb5cc53ec9","Type":"ContainerStarted","Data":"bdc43944908e1b3f57ce1ffa50d21be9dbf1b75e243c08063bfd6a655105f89f"} Sep 11 19:45:23.630378 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:23.630343 2559 generic.go:356] "Generic (PLEG): container finished" podID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerID="bdc43944908e1b3f57ce1ffa50d21be9dbf1b75e243c08063bfd6a655105f89f" exitCode=0 Sep 11 19:45:23.630844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:23.630421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" event={"ID":"d175fbbd-1399-477b-b320-8abb5cc53ec9","Type":"ContainerDied","Data":"bdc43944908e1b3f57ce1ffa50d21be9dbf1b75e243c08063bfd6a655105f89f"} Sep 11 19:45:23.630844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:23.630514 2559 scope.go:117] "RemoveContainer" containerID="69dadf37c199cc6f33d620e67fd426e53234ed8f1413f6c56bd6a8df74ab8cae" Sep 11 19:45:23.665023 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:23.664995 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l"] Sep 11 19:45:24.771090 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:24.771065 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" Sep 11 19:45:25.646150 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:25.646111 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" event={"ID":"d175fbbd-1399-477b-b320-8abb5cc53ec9","Type":"ContainerDied","Data":"4c7da8b3aacc63333cf4703d5a94322796134cbf59f37e45fd02ef71e874ee6b"} Sep 11 19:45:25.646150 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:25.646141 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l" Sep 11 19:45:25.646407 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:25.646161 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c7da8b3aacc63333cf4703d5a94322796134cbf59f37e45fd02ef71e874ee6b" Sep 11 19:45:25.678506 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:25.678468 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l"] Sep 11 19:45:25.685091 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:25.685063 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-10cc24-29819265-82j9l"] Sep 11 19:45:27.224396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:27.224358 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" path="/var/lib/kubelet/pods/d175fbbd-1399-477b-b320-8abb5cc53ec9/volumes" Sep 11 19:45:28.092193 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.092116 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz"] Sep 11 19:45:28.092423 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.092377 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" podUID="86938d9a-1cdb-456a-9318-ba427f01b68e" containerName="payload-pre-processing" containerID="cri-o://02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562" gracePeriod=30 Sep 11 19:45:28.344639 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.344580 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:28.438095 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.438067 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm6mw\" (UniqueName: \"kubernetes.io/projected/86938d9a-1cdb-456a-9318-ba427f01b68e-kube-api-access-gm6mw\") pod \"86938d9a-1cdb-456a-9318-ba427f01b68e\" (UID: \"86938d9a-1cdb-456a-9318-ba427f01b68e\") " Sep 11 19:45:28.438310 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.438128 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86938d9a-1cdb-456a-9318-ba427f01b68e-plugins-config-volume\") pod \"86938d9a-1cdb-456a-9318-ba427f01b68e\" (UID: \"86938d9a-1cdb-456a-9318-ba427f01b68e\") " Sep 11 19:45:28.438600 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.438573 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86938d9a-1cdb-456a-9318-ba427f01b68e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86938d9a-1cdb-456a-9318-ba427f01b68e" (UID: "86938d9a-1cdb-456a-9318-ba427f01b68e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:45:28.440494 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.440404 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86938d9a-1cdb-456a-9318-ba427f01b68e-kube-api-access-gm6mw" (OuterVolumeSpecName: "kube-api-access-gm6mw") pod "86938d9a-1cdb-456a-9318-ba427f01b68e" (UID: "86938d9a-1cdb-456a-9318-ba427f01b68e"). InnerVolumeSpecName "kube-api-access-gm6mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:45:28.486719 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.486687 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf"] Sep 11 19:45:28.491302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.491265 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4c47b5e2-vm9vf"] Sep 11 19:45:28.539213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.539182 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6mw\" (UniqueName: \"kubernetes.io/projected/86938d9a-1cdb-456a-9318-ba427f01b68e-kube-api-access-gm6mw\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:45:28.539213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.539209 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86938d9a-1cdb-456a-9318-ba427f01b68e-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:45:28.594539 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.594509 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:45:28.594746 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.594576 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:45:28.668397 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.668362 2559 generic.go:356] "Generic (PLEG): container finished" podID="86938d9a-1cdb-456a-9318-ba427f01b68e" containerID="02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562" exitCode=0 Sep 11 19:45:28.668586 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.668406 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" Sep 11 19:45:28.668586 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.668465 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" event={"ID":"86938d9a-1cdb-456a-9318-ba427f01b68e","Type":"ContainerDied","Data":"02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562"} Sep 11 19:45:28.668586 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.668533 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz" event={"ID":"86938d9a-1cdb-456a-9318-ba427f01b68e","Type":"ContainerDied","Data":"ca123b20dd3b2ca4c16800e6224eaa7ce317fefbb78866738efb71bb4e7cced6"} Sep 11 19:45:28.668586 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.668565 2559 scope.go:117] "RemoveContainer" containerID="02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562" Sep 11 19:45:28.678798 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.678781 2559 scope.go:117] "RemoveContainer" containerID="02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562" Sep 11 19:45:28.679130 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:45:28.679111 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562\": container with ID starting with 02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562 not found: ID does not exist" containerID="02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562" Sep 11 19:45:28.679204 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.679141 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562"} err="failed to get container status \"02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562\": rpc error: code = NotFound desc = could not find container \"02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562\": container with ID starting with 02a179d941afd3f27e52eb36cdaf68abed2dc1ec3a7292e15f03787f2e5db562 not found: ID does not exist" Sep 11 19:45:28.691279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.691248 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz"] Sep 11 19:45:28.695813 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:28.695788 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4c47b5e2-799c778c46-4sjgz"] Sep 11 19:45:29.224209 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:29.224174 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86938d9a-1cdb-456a-9318-ba427f01b68e" path="/var/lib/kubelet/pods/86938d9a-1cdb-456a-9318-ba427f01b68e/volumes" Sep 11 19:45:29.224534 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:29.224520 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec643558-a83e-4960-b646-ace228c51be0" path="/var/lib/kubelet/pods/ec643558-a83e-4960-b646-ace228c51be0/volumes" Sep 11 19:45:31.879925 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:31.879883 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:45:36.571198 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571153 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx"] Sep 11 19:45:36.571674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571599 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.571674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571612 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.571674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571629 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec643558-a83e-4960-b646-ace228c51be0" containerName="revoke-keys" Sep 11 19:45:36.571674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571635 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec643558-a83e-4960-b646-ace228c51be0" containerName="revoke-keys" Sep 11 19:45:36.571674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571649 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.571674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571654 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.571674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571665 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86938d9a-1cdb-456a-9318-ba427f01b68e" containerName="payload-pre-processing" Sep 11 19:45:36.571674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571670 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="86938d9a-1cdb-456a-9318-ba427f01b68e" containerName="payload-pre-processing" Sep 11 19:45:36.572008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571733 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.572008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571742 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="86938d9a-1cdb-456a-9318-ba427f01b68e" containerName="payload-pre-processing" Sep 11 19:45:36.572008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571749 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec643558-a83e-4960-b646-ace228c51be0" containerName="revoke-keys" Sep 11 19:45:36.572008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.571756 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.575027 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.575011 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:36.578136 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.578106 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-2816752e-dockercfg-5425t\"" Sep 11 19:45:36.578341 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.578327 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-2816752e\"" Sep 11 19:45:36.594499 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.594471 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx"] Sep 11 19:45:36.613108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.613079 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa017a8-bf83-4762-afa0-e134dee9010a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx\" (UID: \"9aa017a8-bf83-4762-afa0-e134dee9010a\") " pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:36.613217 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.613130 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggjhv\" (UniqueName: \"kubernetes.io/projected/9aa017a8-bf83-4762-afa0-e134dee9010a-kube-api-access-ggjhv\") pod \"payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx\" (UID: \"9aa017a8-bf83-4762-afa0-e134dee9010a\") " pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:36.636409 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.636379 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt"] Sep 11 19:45:36.636795 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.636782 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.636838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.636798 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.636885 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.636876 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d175fbbd-1399-477b-b320-8abb5cc53ec9" containerName="cleanup" Sep 11 19:45:36.641495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.641479 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:36.650755 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.650734 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt"] Sep 11 19:45:36.714333 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.714298 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa017a8-bf83-4762-afa0-e134dee9010a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx\" (UID: \"9aa017a8-bf83-4762-afa0-e134dee9010a\") " pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:36.714534 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.714360 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggjhv\" (UniqueName: \"kubernetes.io/projected/9aa017a8-bf83-4762-afa0-e134dee9010a-kube-api-access-ggjhv\") pod \"payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx\" (UID: \"9aa017a8-bf83-4762-afa0-e134dee9010a\") " pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:36.714534 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.714392 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517b10f9-90df-4848-8347-b902639d83ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt\" (UID: \"517b10f9-90df-4848-8347-b902639d83ed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:36.714534 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.714470 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjg72\" (UniqueName: \"kubernetes.io/projected/517b10f9-90df-4848-8347-b902639d83ed-kube-api-access-pjg72\") pod \"payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt\" (UID: \"517b10f9-90df-4848-8347-b902639d83ed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:36.715105 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.715079 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa017a8-bf83-4762-afa0-e134dee9010a-plugins-config-volume\") pod \"payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx\" (UID: \"9aa017a8-bf83-4762-afa0-e134dee9010a\") " pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:36.726448 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.726405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggjhv\" (UniqueName: \"kubernetes.io/projected/9aa017a8-bf83-4762-afa0-e134dee9010a-kube-api-access-ggjhv\") pod \"payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx\" (UID: \"9aa017a8-bf83-4762-afa0-e134dee9010a\") " pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:36.815855 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.815815 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517b10f9-90df-4848-8347-b902639d83ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt\" (UID: \"517b10f9-90df-4848-8347-b902639d83ed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:36.816027 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.815894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjg72\" (UniqueName: \"kubernetes.io/projected/517b10f9-90df-4848-8347-b902639d83ed-kube-api-access-pjg72\") pod \"payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt\" (UID: \"517b10f9-90df-4848-8347-b902639d83ed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:36.816546 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.816526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517b10f9-90df-4848-8347-b902639d83ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt\" (UID: \"517b10f9-90df-4848-8347-b902639d83ed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:36.824532 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.824464 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjg72\" (UniqueName: \"kubernetes.io/projected/517b10f9-90df-4848-8347-b902639d83ed-kube-api-access-pjg72\") pod \"payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt\" (UID: \"517b10f9-90df-4848-8347-b902639d83ed\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:36.886495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.886461 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:36.953096 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:36.952778 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:37.168042 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.168014 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx"] Sep 11 19:45:37.169097 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:37.169073 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa017a8_bf83_4762_afa0_e134dee9010a.slice/crio-e550432f4d702630b2cbc6c396273142d3cd82bbf5b8014d21c61f8d7afcf86b WatchSource:0}: Error finding container e550432f4d702630b2cbc6c396273142d3cd82bbf5b8014d21c61f8d7afcf86b: Status 404 returned error can't find the container with id e550432f4d702630b2cbc6c396273142d3cd82bbf5b8014d21c61f8d7afcf86b Sep 11 19:45:37.234900 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.234877 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt"] Sep 11 19:45:37.237144 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:45:37.237121 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517b10f9_90df_4848_8347_b902639d83ed.slice/crio-2af7af6a3ddaf4f453ef2a1067067d2ee745dcda719c491d687052360c4e6f87 WatchSource:0}: Error finding container 2af7af6a3ddaf4f453ef2a1067067d2ee745dcda719c491d687052360c4e6f87: Status 404 returned error can't find the container with id 2af7af6a3ddaf4f453ef2a1067067d2ee745dcda719c491d687052360c4e6f87 Sep 11 19:45:37.729061 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.729021 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" event={"ID":"517b10f9-90df-4848-8347-b902639d83ed","Type":"ContainerStarted","Data":"cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066"} Sep 11 19:45:37.729607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.729098 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" event={"ID":"517b10f9-90df-4848-8347-b902639d83ed","Type":"ContainerStarted","Data":"2af7af6a3ddaf4f453ef2a1067067d2ee745dcda719c491d687052360c4e6f87"} Sep 11 19:45:37.729607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.729237 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:37.731475 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.731430 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" event={"ID":"9aa017a8-bf83-4762-afa0-e134dee9010a","Type":"ContainerStarted","Data":"114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced"} Sep 11 19:45:37.731595 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.731497 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" event={"ID":"9aa017a8-bf83-4762-afa0-e134dee9010a","Type":"ContainerStarted","Data":"e550432f4d702630b2cbc6c396273142d3cd82bbf5b8014d21c61f8d7afcf86b"} Sep 11 19:45:37.731595 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.731569 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:37.752296 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.752244 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" podStartSLOduration=1.752226029 podStartE2EDuration="1.752226029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:37.748653185 +0000 UTC m=+1171.021160384" watchObservedRunningTime="2026-09-11 19:45:37.752226029 +0000 UTC m=+1171.024733226" Sep 11 19:45:37.770800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:37.770757 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" podStartSLOduration=1.770741411 podStartE2EDuration="1.770741411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:45:37.769391435 +0000 UTC m=+1171.041898639" watchObservedRunningTime="2026-09-11 19:45:37.770741411 +0000 UTC m=+1171.043248608" Sep 11 19:45:48.740232 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:48.740200 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:45:48.740678 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:48.740453 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:45:49.080517 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:49.080423 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:45:59.447864 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.447829 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f"] Sep 11 19:45:59.453149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.453130 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.457236 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.457216 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"kube-root-ca.crt\"" Sep 11 19:45:59.457350 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.457214 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"openshift-service-ca.crt\"" Sep 11 19:45:59.457350 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.457305 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"simulator-w4-c7766c-kserve-self-signed-certs\"" Sep 11 19:45:59.457493 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.457374 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c7766c\"/\"default-dockercfg-vxzf9\"" Sep 11 19:45:59.461944 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.461926 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f"] Sep 11 19:45:59.540087 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.540054 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-dshm\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.540279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.540116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-home\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.540279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.540160 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzv6f\" (UniqueName: \"kubernetes.io/projected/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-kube-api-access-jzv6f\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.540279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.540213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tmp-dir\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.540279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.540236 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tls-certs\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.540279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.540278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-model-cache\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641080 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641040 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-home\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641080 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641081 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzv6f\" (UniqueName: \"kubernetes.io/projected/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-kube-api-access-jzv6f\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tmp-dir\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641145 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tls-certs\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-model-cache\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-dshm\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641601 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-home\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641601 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641583 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-model-cache\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.641691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.641622 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tmp-dir\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.643672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.643645 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-dshm\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.643928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.643909 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tls-certs\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.650837 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.650813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzv6f\" (UniqueName: \"kubernetes.io/projected/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-kube-api-access-jzv6f\") pod \"simulator-w4-c7766c-kserve-7db674b865-lxf2f\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:45:59.764189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:45:59.764147 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:46:00.043541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:00.043517 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f"] Sep 11 19:46:00.044176 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:46:00.044146 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ec30a3e_46f7_4a6a_9d25_824a9bcbb129.slice/crio-48ca4b82f056d33e57fb2b5746328c9d04cea8c53ef6dc7ff3dfeb1567d22f1c WatchSource:0}: Error finding container 48ca4b82f056d33e57fb2b5746328c9d04cea8c53ef6dc7ff3dfeb1567d22f1c: Status 404 returned error can't find the container with id 48ca4b82f056d33e57fb2b5746328c9d04cea8c53ef6dc7ff3dfeb1567d22f1c Sep 11 19:46:00.884992 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:00.884959 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" event={"ID":"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129","Type":"ContainerStarted","Data":"f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7"} Sep 11 19:46:00.885370 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:00.884998 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" event={"ID":"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129","Type":"ContainerStarted","Data":"48ca4b82f056d33e57fb2b5746328c9d04cea8c53ef6dc7ff3dfeb1567d22f1c"} Sep 11 19:46:00.902788 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:00.902743 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" podStartSLOduration=1.902726167 podStartE2EDuration="1.902726167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:00.901229562 +0000 UTC m=+1194.173736757" watchObservedRunningTime="2026-09-11 19:46:00.902726167 +0000 UTC m=+1194.175233369" Sep 11 19:46:02.395448 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.395396 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2"] Sep 11 19:46:02.399301 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.399268 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" Sep 11 19:46:02.402145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.402112 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:46:02.402406 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.402389 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:46:02.411759 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.411733 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2"] Sep 11 19:46:02.570066 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.570024 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b924ad3-5d63-473d-b4a2-0b9876f6e90b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2\" (UID: \"5b924ad3-5d63-473d-b4a2-0b9876f6e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" Sep 11 19:46:02.671765 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.671677 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b924ad3-5d63-473d-b4a2-0b9876f6e90b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2\" (UID: \"5b924ad3-5d63-473d-b4a2-0b9876f6e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" Sep 11 19:46:02.672330 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.672308 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b924ad3-5d63-473d-b4a2-0b9876f6e90b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2\" (UID: \"5b924ad3-5d63-473d-b4a2-0b9876f6e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" Sep 11 19:46:02.712514 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.712476 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" Sep 11 19:46:02.996281 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:02.996246 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2"] Sep 11 19:46:02.997611 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:46:02.997578 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b924ad3_5d63_473d_b4a2_0b9876f6e90b.slice/crio-353718a1ff4ec5cad79a4f00e1046da66dfe1fc45aa6c674318e4a84aef94e90 WatchSource:0}: Error finding container 353718a1ff4ec5cad79a4f00e1046da66dfe1fc45aa6c674318e4a84aef94e90: Status 404 returned error can't find the container with id 353718a1ff4ec5cad79a4f00e1046da66dfe1fc45aa6c674318e4a84aef94e90 Sep 11 19:46:03.908349 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:03.908315 2559 generic.go:356] "Generic (PLEG): container finished" podID="5b924ad3-5d63-473d-b4a2-0b9876f6e90b" containerID="56e6e7b2a9099871a54135ffbc8b7365c2fd79cf54967f2ab513af13d37395f4" exitCode=0 Sep 11 19:46:03.908817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:03.908355 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" event={"ID":"5b924ad3-5d63-473d-b4a2-0b9876f6e90b","Type":"ContainerDied","Data":"56e6e7b2a9099871a54135ffbc8b7365c2fd79cf54967f2ab513af13d37395f4"} Sep 11 19:46:03.908817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:03.908415 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" event={"ID":"5b924ad3-5d63-473d-b4a2-0b9876f6e90b","Type":"ContainerStarted","Data":"353718a1ff4ec5cad79a4f00e1046da66dfe1fc45aa6c674318e4a84aef94e90"} Sep 11 19:46:04.925752 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:04.925720 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:46:05.072928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:05.072902 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" Sep 11 19:46:05.201088 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:05.201000 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b924ad3-5d63-473d-b4a2-0b9876f6e90b-maas-api-service-ca\") pod \"5b924ad3-5d63-473d-b4a2-0b9876f6e90b\" (UID: \"5b924ad3-5d63-473d-b4a2-0b9876f6e90b\") " Sep 11 19:46:05.201410 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:05.201384 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b924ad3-5d63-473d-b4a2-0b9876f6e90b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b924ad3-5d63-473d-b4a2-0b9876f6e90b" (UID: "5b924ad3-5d63-473d-b4a2-0b9876f6e90b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:46:05.302473 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:05.302409 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b924ad3-5d63-473d-b4a2-0b9876f6e90b-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:05.923529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:05.923487 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" event={"ID":"5b924ad3-5d63-473d-b4a2-0b9876f6e90b","Type":"ContainerDied","Data":"353718a1ff4ec5cad79a4f00e1046da66dfe1fc45aa6c674318e4a84aef94e90"} Sep 11 19:46:05.923690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:05.923563 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="353718a1ff4ec5cad79a4f00e1046da66dfe1fc45aa6c674318e4a84aef94e90" Sep 11 19:46:05.923690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:05.923499 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2" Sep 11 19:46:09.764319 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:09.764283 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:46:09.764805 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:09.764332 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:46:09.776921 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:09.776887 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:46:09.968879 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:09.968843 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:46:14.370426 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.370382 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx"] Sep 11 19:46:14.370936 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.370713 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" podUID="9aa017a8-bf83-4762-afa0-e134dee9010a" containerName="payload-processing" containerID="cri-o://114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced" gracePeriod=30 Sep 11 19:46:14.381319 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.381291 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt"] Sep 11 19:46:14.381570 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.381540 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" podUID="517b10f9-90df-4848-8347-b902639d83ed" containerName="payload-pre-processing" containerID="cri-o://cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066" gracePeriod=30 Sep 11 19:46:14.619508 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.619470 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2"] Sep 11 19:46:14.624106 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.624039 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-2816752e-7dzd2"] Sep 11 19:46:14.667579 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.667552 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:46:14.670634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.670612 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:46:14.680987 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.680966 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjg72\" (UniqueName: \"kubernetes.io/projected/517b10f9-90df-4848-8347-b902639d83ed-kube-api-access-pjg72\") pod \"517b10f9-90df-4848-8347-b902639d83ed\" (UID: \"517b10f9-90df-4848-8347-b902639d83ed\") " Sep 11 19:46:14.681061 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.681031 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ggjhv\" (UniqueName: \"kubernetes.io/projected/9aa017a8-bf83-4762-afa0-e134dee9010a-kube-api-access-ggjhv\") pod \"9aa017a8-bf83-4762-afa0-e134dee9010a\" (UID: \"9aa017a8-bf83-4762-afa0-e134dee9010a\") " Sep 11 19:46:14.681100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.681070 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa017a8-bf83-4762-afa0-e134dee9010a-plugins-config-volume\") pod \"9aa017a8-bf83-4762-afa0-e134dee9010a\" (UID: \"9aa017a8-bf83-4762-afa0-e134dee9010a\") " Sep 11 19:46:14.681133 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.681103 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517b10f9-90df-4848-8347-b902639d83ed-plugins-config-volume\") pod \"517b10f9-90df-4848-8347-b902639d83ed\" (UID: \"517b10f9-90df-4848-8347-b902639d83ed\") " Sep 11 19:46:14.681425 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.681394 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aa017a8-bf83-4762-afa0-e134dee9010a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9aa017a8-bf83-4762-afa0-e134dee9010a" (UID: "9aa017a8-bf83-4762-afa0-e134dee9010a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:46:14.681523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.681504 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517b10f9-90df-4848-8347-b902639d83ed-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "517b10f9-90df-4848-8347-b902639d83ed" (UID: "517b10f9-90df-4848-8347-b902639d83ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:46:14.683228 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.683207 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa017a8-bf83-4762-afa0-e134dee9010a-kube-api-access-ggjhv" (OuterVolumeSpecName: "kube-api-access-ggjhv") pod "9aa017a8-bf83-4762-afa0-e134dee9010a" (UID: "9aa017a8-bf83-4762-afa0-e134dee9010a"). InnerVolumeSpecName "kube-api-access-ggjhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:46:14.683302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.683282 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517b10f9-90df-4848-8347-b902639d83ed-kube-api-access-pjg72" (OuterVolumeSpecName: "kube-api-access-pjg72") pod "517b10f9-90df-4848-8347-b902639d83ed" (UID: "517b10f9-90df-4848-8347-b902639d83ed"). InnerVolumeSpecName "kube-api-access-pjg72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:46:14.782034 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.782001 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggjhv\" (UniqueName: \"kubernetes.io/projected/9aa017a8-bf83-4762-afa0-e134dee9010a-kube-api-access-ggjhv\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:14.782034 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.782028 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aa017a8-bf83-4762-afa0-e134dee9010a-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:14.782034 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.782037 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/517b10f9-90df-4848-8347-b902639d83ed-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:14.782269 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.782046 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjg72\" (UniqueName: \"kubernetes.io/projected/517b10f9-90df-4848-8347-b902639d83ed-kube-api-access-pjg72\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:14.991166 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.991135 2559 generic.go:356] "Generic (PLEG): container finished" podID="517b10f9-90df-4848-8347-b902639d83ed" containerID="cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066" exitCode=0 Sep 11 19:46:14.991357 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.991178 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" Sep 11 19:46:14.991357 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.991227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" event={"ID":"517b10f9-90df-4848-8347-b902639d83ed","Type":"ContainerDied","Data":"cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066"} Sep 11 19:46:14.991357 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.991279 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt" event={"ID":"517b10f9-90df-4848-8347-b902639d83ed","Type":"ContainerDied","Data":"2af7af6a3ddaf4f453ef2a1067067d2ee745dcda719c491d687052360c4e6f87"} Sep 11 19:46:14.991357 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.991301 2559 scope.go:117] "RemoveContainer" containerID="cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066" Sep 11 19:46:14.992838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.992815 2559 generic.go:356] "Generic (PLEG): container finished" podID="9aa017a8-bf83-4762-afa0-e134dee9010a" containerID="114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced" exitCode=0 Sep 11 19:46:14.992968 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.992868 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" event={"ID":"9aa017a8-bf83-4762-afa0-e134dee9010a","Type":"ContainerDied","Data":"114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced"} Sep 11 19:46:14.992968 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.992881 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" Sep 11 19:46:14.992968 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:14.992911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx" event={"ID":"9aa017a8-bf83-4762-afa0-e134dee9010a","Type":"ContainerDied","Data":"e550432f4d702630b2cbc6c396273142d3cd82bbf5b8014d21c61f8d7afcf86b"} Sep 11 19:46:15.002868 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.002814 2559 scope.go:117] "RemoveContainer" containerID="cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066" Sep 11 19:46:15.003188 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:46:15.003167 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066\": container with ID starting with cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066 not found: ID does not exist" containerID="cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066" Sep 11 19:46:15.003286 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.003200 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066"} err="failed to get container status \"cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066\": rpc error: code = NotFound desc = could not find container \"cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066\": container with ID starting with cd4bec35787bf777d02823072c12198b23fc891d65ee9118c09997a5fa028066 not found: ID does not exist" Sep 11 19:46:15.003286 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.003224 2559 scope.go:117] "RemoveContainer" containerID="114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced" Sep 11 19:46:15.012870 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.012853 2559 scope.go:117] "RemoveContainer" containerID="114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced" Sep 11 19:46:15.013387 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:46:15.013324 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced\": container with ID starting with 114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced not found: ID does not exist" containerID="114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced" Sep 11 19:46:15.013387 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.013357 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced"} err="failed to get container status \"114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced\": rpc error: code = NotFound desc = could not find container \"114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced\": container with ID starting with 114546a8f3a064dd993c613d2479892244fbe5a69d8ec80d9f71d068eefbfced not found: ID does not exist" Sep 11 19:46:15.021711 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.021688 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx"] Sep 11 19:46:15.028131 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.028110 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-2816752e-7d48dbfbbb-8rzgx"] Sep 11 19:46:15.040643 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.040624 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt"] Sep 11 19:46:15.049528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.049496 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-2816752e-7b7c864858-czslt"] Sep 11 19:46:15.224900 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.224869 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="517b10f9-90df-4848-8347-b902639d83ed" path="/var/lib/kubelet/pods/517b10f9-90df-4848-8347-b902639d83ed/volumes" Sep 11 19:46:15.225206 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.225194 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b924ad3-5d63-473d-b4a2-0b9876f6e90b" path="/var/lib/kubelet/pods/5b924ad3-5d63-473d-b4a2-0b9876f6e90b/volumes" Sep 11 19:46:15.225517 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:15.225504 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aa017a8-bf83-4762-afa0-e134dee9010a" path="/var/lib/kubelet/pods/9aa017a8-bf83-4762-afa0-e134dee9010a/volumes" Sep 11 19:46:19.082844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:19.082807 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:46:21.124862 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.124807 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2"] Sep 11 19:46:21.125374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125286 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aa017a8-bf83-4762-afa0-e134dee9010a" containerName="payload-processing" Sep 11 19:46:21.125374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125303 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aa017a8-bf83-4762-afa0-e134dee9010a" containerName="payload-processing" Sep 11 19:46:21.125374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125317 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="517b10f9-90df-4848-8347-b902639d83ed" containerName="payload-pre-processing" Sep 11 19:46:21.125374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125324 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="517b10f9-90df-4848-8347-b902639d83ed" containerName="payload-pre-processing" Sep 11 19:46:21.125374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125337 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b924ad3-5d63-473d-b4a2-0b9876f6e90b" containerName="revoke-keys" Sep 11 19:46:21.125374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125343 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b924ad3-5d63-473d-b4a2-0b9876f6e90b" containerName="revoke-keys" Sep 11 19:46:21.125633 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125426 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aa017a8-bf83-4762-afa0-e134dee9010a" containerName="payload-processing" Sep 11 19:46:21.125633 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125449 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b924ad3-5d63-473d-b4a2-0b9876f6e90b" containerName="revoke-keys" Sep 11 19:46:21.125633 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.125462 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="517b10f9-90df-4848-8347-b902639d83ed" containerName="payload-pre-processing" Sep 11 19:46:21.130048 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.130030 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:21.133653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.133634 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8901a783-dockercfg-fjfkj\"" Sep 11 19:46:21.133778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.133713 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8901a783\"" Sep 11 19:46:21.143371 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.143347 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2"] Sep 11 19:46:21.195998 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.195962 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822"] Sep 11 19:46:21.199616 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.199595 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:21.211672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.211646 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822"] Sep 11 19:46:21.239016 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.238979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7lj\" (UniqueName: \"kubernetes.io/projected/61674e54-e1a7-48cc-8eac-00b083ba49cc-kube-api-access-zm7lj\") pod \"payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2\" (UID: \"61674e54-e1a7-48cc-8eac-00b083ba49cc\") " pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:21.239203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.239028 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61674e54-e1a7-48cc-8eac-00b083ba49cc-plugins-config-volume\") pod \"payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2\" (UID: \"61674e54-e1a7-48cc-8eac-00b083ba49cc\") " pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:21.239203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.239087 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tts2q\" (UniqueName: \"kubernetes.io/projected/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-kube-api-access-tts2q\") pod \"payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822\" (UID: \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:21.239323 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.239238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822\" (UID: \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:21.340514 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.340477 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822\" (UID: \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:21.340703 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.340530 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7lj\" (UniqueName: \"kubernetes.io/projected/61674e54-e1a7-48cc-8eac-00b083ba49cc-kube-api-access-zm7lj\") pod \"payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2\" (UID: \"61674e54-e1a7-48cc-8eac-00b083ba49cc\") " pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:21.340703 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.340573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61674e54-e1a7-48cc-8eac-00b083ba49cc-plugins-config-volume\") pod \"payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2\" (UID: \"61674e54-e1a7-48cc-8eac-00b083ba49cc\") " pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:21.340703 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.340611 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tts2q\" (UniqueName: \"kubernetes.io/projected/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-kube-api-access-tts2q\") pod \"payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822\" (UID: \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:21.341285 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.341265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61674e54-e1a7-48cc-8eac-00b083ba49cc-plugins-config-volume\") pod \"payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2\" (UID: \"61674e54-e1a7-48cc-8eac-00b083ba49cc\") " pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:21.341342 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.341265 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822\" (UID: \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:21.351694 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.351670 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tts2q\" (UniqueName: \"kubernetes.io/projected/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-kube-api-access-tts2q\") pod \"payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822\" (UID: \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:21.351832 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.351697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7lj\" (UniqueName: \"kubernetes.io/projected/61674e54-e1a7-48cc-8eac-00b083ba49cc-kube-api-access-zm7lj\") pod \"payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2\" (UID: \"61674e54-e1a7-48cc-8eac-00b083ba49cc\") " pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:21.441188 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.441089 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:21.510758 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.510728 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:21.716472 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.716428 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2"] Sep 11 19:46:21.718677 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:46:21.718651 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61674e54_e1a7_48cc_8eac_00b083ba49cc.slice/crio-0a51deb30a45d2c578c39f87a57585887f7f5620258ec7e0306f8b35a364b3d7 WatchSource:0}: Error finding container 0a51deb30a45d2c578c39f87a57585887f7f5620258ec7e0306f8b35a364b3d7: Status 404 returned error can't find the container with id 0a51deb30a45d2c578c39f87a57585887f7f5620258ec7e0306f8b35a364b3d7 Sep 11 19:46:21.791835 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:21.791800 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822"] Sep 11 19:46:21.793133 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:46:21.793103 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa71e8b2_0626_4d76_87ac_32bbef8ba61c.slice/crio-cb7034b1f0bfd8e659d043ed91aa8fec374bb41cff940449f18c44db0eb572b9 WatchSource:0}: Error finding container cb7034b1f0bfd8e659d043ed91aa8fec374bb41cff940449f18c44db0eb572b9: Status 404 returned error can't find the container with id cb7034b1f0bfd8e659d043ed91aa8fec374bb41cff940449f18c44db0eb572b9 Sep 11 19:46:22.049606 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:22.049560 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" event={"ID":"fa71e8b2-0626-4d76-87ac-32bbef8ba61c","Type":"ContainerStarted","Data":"b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87"} Sep 11 19:46:22.049822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:22.049621 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" event={"ID":"fa71e8b2-0626-4d76-87ac-32bbef8ba61c","Type":"ContainerStarted","Data":"cb7034b1f0bfd8e659d043ed91aa8fec374bb41cff940449f18c44db0eb572b9"} Sep 11 19:46:22.049822 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:22.049771 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:22.051380 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:22.051347 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" event={"ID":"61674e54-e1a7-48cc-8eac-00b083ba49cc","Type":"ContainerStarted","Data":"080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db"} Sep 11 19:46:22.051529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:22.051397 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" event={"ID":"61674e54-e1a7-48cc-8eac-00b083ba49cc","Type":"ContainerStarted","Data":"0a51deb30a45d2c578c39f87a57585887f7f5620258ec7e0306f8b35a364b3d7"} Sep 11 19:46:22.051529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:22.051428 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:22.068505 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:22.068463 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" podStartSLOduration=1.068421821 podStartE2EDuration="1.068421821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:22.06722159 +0000 UTC m=+1215.339728793" watchObservedRunningTime="2026-09-11 19:46:22.068421821 +0000 UTC m=+1215.340929020" Sep 11 19:46:22.087045 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:22.087007 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" podStartSLOduration=1.086993588 podStartE2EDuration="1.086993588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:22.084085454 +0000 UTC m=+1215.356592656" watchObservedRunningTime="2026-09-11 19:46:22.086993588 +0000 UTC m=+1215.359500834" Sep 11 19:46:33.060637 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:33.060597 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:33.063177 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:33.060739 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:40.762667 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:40.762636 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:46:41.171592 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.171516 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj"] Sep 11 19:46:41.174906 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.174883 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.179257 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.179235 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"kube-root-ca.crt\"" Sep 11 19:46:41.179386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.179272 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"simulator-w0-10cc24-kserve-self-signed-certs\"" Sep 11 19:46:41.179545 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.179526 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"default-dockercfg-z9db5\"" Sep 11 19:46:41.179647 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.179615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10cc24\"/\"openshift-service-ca.crt\"" Sep 11 19:46:41.191848 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.191824 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj"] Sep 11 19:46:41.347139 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.347103 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-home\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.347139 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.347144 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-tmp-dir\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.347386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.347251 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-model-cache\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.347386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.347285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npq8v\" (UniqueName: \"kubernetes.io/projected/9f977182-c120-4088-ba89-e300e3d347d0-kube-api-access-npq8v\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.347535 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.347421 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f977182-c120-4088-ba89-e300e3d347d0-tls-certs\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.347535 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.347478 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-dshm\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.448766 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.448678 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-home\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.448766 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.448716 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-tmp-dir\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.448999 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.448769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-model-cache\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.448999 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.448798 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npq8v\" (UniqueName: \"kubernetes.io/projected/9f977182-c120-4088-ba89-e300e3d347d0-kube-api-access-npq8v\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.448999 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.448907 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f977182-c120-4088-ba89-e300e3d347d0-tls-certs\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.448999 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.448934 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-dshm\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.449198 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.449090 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-model-cache\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.449198 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.449131 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-home\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.449198 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.449170 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-tmp-dir\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.451394 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.451373 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-dshm\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.451556 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.451540 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f977182-c120-4088-ba89-e300e3d347d0-tls-certs\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.457995 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.457971 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npq8v\" (UniqueName: \"kubernetes.io/projected/9f977182-c120-4088-ba89-e300e3d347d0-kube-api-access-npq8v\") pod \"simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.484992 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.484966 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:41.763834 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:41.763801 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj"] Sep 11 19:46:41.764267 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:46:41.764241 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f977182_c120_4088_ba89_e300e3d347d0.slice/crio-8f5eb9d6dba0a2b5f7bf2e14d79549e339d4cc0e61a0e985f480975871c287a4 WatchSource:0}: Error finding container 8f5eb9d6dba0a2b5f7bf2e14d79549e339d4cc0e61a0e985f480975871c287a4: Status 404 returned error can't find the container with id 8f5eb9d6dba0a2b5f7bf2e14d79549e339d4cc0e61a0e985f480975871c287a4 Sep 11 19:46:42.186504 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:42.186389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" event={"ID":"9f977182-c120-4088-ba89-e300e3d347d0","Type":"ContainerStarted","Data":"dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b"} Sep 11 19:46:42.186504 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:42.186464 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" event={"ID":"9f977182-c120-4088-ba89-e300e3d347d0","Type":"ContainerStarted","Data":"8f5eb9d6dba0a2b5f7bf2e14d79549e339d4cc0e61a0e985f480975871c287a4"} Sep 11 19:46:42.208984 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:42.208930 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" podStartSLOduration=1.208913573 podStartE2EDuration="1.208913573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:42.207344092 +0000 UTC m=+1235.479851296" watchObservedRunningTime="2026-09-11 19:46:42.208913573 +0000 UTC m=+1235.481420773" Sep 11 19:46:44.184979 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.184946 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79"] Sep 11 19:46:44.188537 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.188520 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" Sep 11 19:46:44.191919 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.191891 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:46:44.192308 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.192020 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:46:44.203935 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.201099 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79"] Sep 11 19:46:44.275980 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.275943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03986d10-3a98-4e19-90b4-ac685d579f61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8901a783-qpz79\" (UID: \"03986d10-3a98-4e19-90b4-ac685d579f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" Sep 11 19:46:44.377249 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.377210 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03986d10-3a98-4e19-90b4-ac685d579f61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8901a783-qpz79\" (UID: \"03986d10-3a98-4e19-90b4-ac685d579f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" Sep 11 19:46:44.377916 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.377894 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03986d10-3a98-4e19-90b4-ac685d579f61-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8901a783-qpz79\" (UID: \"03986d10-3a98-4e19-90b4-ac685d579f61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" Sep 11 19:46:44.511952 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.511918 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" Sep 11 19:46:44.801317 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:44.801286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79"] Sep 11 19:46:44.802989 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:46:44.802962 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03986d10_3a98_4e19_90b4_ac685d579f61.slice/crio-d040f7931a9c7d4ec8bf7774b4437ec69b2677733447cebc3c5811503ce012aa WatchSource:0}: Error finding container d040f7931a9c7d4ec8bf7774b4437ec69b2677733447cebc3c5811503ce012aa: Status 404 returned error can't find the container with id d040f7931a9c7d4ec8bf7774b4437ec69b2677733447cebc3c5811503ce012aa Sep 11 19:46:45.211106 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:45.211068 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" event={"ID":"03986d10-3a98-4e19-90b4-ac685d579f61","Type":"ContainerStarted","Data":"d68c3a8de3dd0e848b2df1f07a1594449012e0c567d908645c7c88f245503ee1"} Sep 11 19:46:45.211522 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:45.211126 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" event={"ID":"03986d10-3a98-4e19-90b4-ac685d579f61","Type":"ContainerStarted","Data":"d040f7931a9c7d4ec8bf7774b4437ec69b2677733447cebc3c5811503ce012aa"} Sep 11 19:46:45.228374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:45.228325 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" podStartSLOduration=1.228307043 podStartE2EDuration="1.228307043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:46:45.226283949 +0000 UTC m=+1238.498791149" watchObservedRunningTime="2026-09-11 19:46:45.228307043 +0000 UTC m=+1238.500814240" Sep 11 19:46:46.219332 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:46.219298 2559 generic.go:356] "Generic (PLEG): container finished" podID="03986d10-3a98-4e19-90b4-ac685d579f61" containerID="d68c3a8de3dd0e848b2df1f07a1594449012e0c567d908645c7c88f245503ee1" exitCode=0 Sep 11 19:46:46.219736 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:46.219365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" event={"ID":"03986d10-3a98-4e19-90b4-ac685d579f61","Type":"ContainerDied","Data":"d68c3a8de3dd0e848b2df1f07a1594449012e0c567d908645c7c88f245503ee1"} Sep 11 19:46:47.365704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:47.365680 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" Sep 11 19:46:47.510552 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:47.510519 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03986d10-3a98-4e19-90b4-ac685d579f61-maas-api-service-ca\") pod \"03986d10-3a98-4e19-90b4-ac685d579f61\" (UID: \"03986d10-3a98-4e19-90b4-ac685d579f61\") " Sep 11 19:46:47.510895 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:47.510871 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03986d10-3a98-4e19-90b4-ac685d579f61-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "03986d10-3a98-4e19-90b4-ac685d579f61" (UID: "03986d10-3a98-4e19-90b4-ac685d579f61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:46:47.612236 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:47.612207 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/03986d10-3a98-4e19-90b4-ac685d579f61-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:48.233970 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:48.233933 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" Sep 11 19:46:48.234118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:48.233970 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79" event={"ID":"03986d10-3a98-4e19-90b4-ac685d579f61","Type":"ContainerDied","Data":"d040f7931a9c7d4ec8bf7774b4437ec69b2677733447cebc3c5811503ce012aa"} Sep 11 19:46:48.234118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:48.234021 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d040f7931a9c7d4ec8bf7774b4437ec69b2677733447cebc3c5811503ce012aa" Sep 11 19:46:51.485542 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:51.485507 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:51.485917 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:51.485762 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:51.497960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:51.497937 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:51.928626 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:51.928593 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2"] Sep 11 19:46:51.928872 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:51.928847 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" podUID="61674e54-e1a7-48cc-8eac-00b083ba49cc" containerName="payload-processing" containerID="cri-o://080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db" gracePeriod=30 Sep 11 19:46:51.943399 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:51.943364 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822"] Sep 11 19:46:51.943844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:51.943804 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" podUID="fa71e8b2-0626-4d76-87ac-32bbef8ba61c" containerName="payload-pre-processing" containerID="cri-o://b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87" gracePeriod=30 Sep 11 19:46:52.179129 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.179043 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79"] Sep 11 19:46:52.184733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.184710 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8901a783-qpz79"] Sep 11 19:46:52.221298 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.221274 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:52.224214 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.224194 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:52.256830 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.256805 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61674e54-e1a7-48cc-8eac-00b083ba49cc-plugins-config-volume\") pod \"61674e54-e1a7-48cc-8eac-00b083ba49cc\" (UID: \"61674e54-e1a7-48cc-8eac-00b083ba49cc\") " Sep 11 19:46:52.257019 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.256844 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zm7lj\" (UniqueName: \"kubernetes.io/projected/61674e54-e1a7-48cc-8eac-00b083ba49cc-kube-api-access-zm7lj\") pod \"61674e54-e1a7-48cc-8eac-00b083ba49cc\" (UID: \"61674e54-e1a7-48cc-8eac-00b083ba49cc\") " Sep 11 19:46:52.257019 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.256921 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tts2q\" (UniqueName: \"kubernetes.io/projected/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-kube-api-access-tts2q\") pod \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\" (UID: \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\") " Sep 11 19:46:52.257739 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.257702 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61674e54-e1a7-48cc-8eac-00b083ba49cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "61674e54-e1a7-48cc-8eac-00b083ba49cc" (UID: "61674e54-e1a7-48cc-8eac-00b083ba49cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:46:52.268084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.259841 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-plugins-config-volume\") pod \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\" (UID: \"fa71e8b2-0626-4d76-87ac-32bbef8ba61c\") " Sep 11 19:46:52.268084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.260494 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61674e54-e1a7-48cc-8eac-00b083ba49cc-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:52.268084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.261145 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa71e8b2-0626-4d76-87ac-32bbef8ba61c" (UID: "fa71e8b2-0626-4d76-87ac-32bbef8ba61c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:46:52.268084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.267452 2559 generic.go:356] "Generic (PLEG): container finished" podID="fa71e8b2-0626-4d76-87ac-32bbef8ba61c" containerID="b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87" exitCode=0 Sep 11 19:46:52.268084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.267561 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" event={"ID":"fa71e8b2-0626-4d76-87ac-32bbef8ba61c","Type":"ContainerDied","Data":"b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87"} Sep 11 19:46:52.268084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.267648 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" event={"ID":"fa71e8b2-0626-4d76-87ac-32bbef8ba61c","Type":"ContainerDied","Data":"cb7034b1f0bfd8e659d043ed91aa8fec374bb41cff940449f18c44db0eb572b9"} Sep 11 19:46:52.268084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.267685 2559 scope.go:117] "RemoveContainer" containerID="b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87" Sep 11 19:46:52.268084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.267880 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822" Sep 11 19:46:52.268853 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.268822 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61674e54-e1a7-48cc-8eac-00b083ba49cc-kube-api-access-zm7lj" (OuterVolumeSpecName: "kube-api-access-zm7lj") pod "61674e54-e1a7-48cc-8eac-00b083ba49cc" (UID: "61674e54-e1a7-48cc-8eac-00b083ba49cc"). InnerVolumeSpecName "kube-api-access-zm7lj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:46:52.269146 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.269111 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-kube-api-access-tts2q" (OuterVolumeSpecName: "kube-api-access-tts2q") pod "fa71e8b2-0626-4d76-87ac-32bbef8ba61c" (UID: "fa71e8b2-0626-4d76-87ac-32bbef8ba61c"). InnerVolumeSpecName "kube-api-access-tts2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:46:52.270783 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.270756 2559 generic.go:356] "Generic (PLEG): container finished" podID="61674e54-e1a7-48cc-8eac-00b083ba49cc" containerID="080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db" exitCode=0 Sep 11 19:46:52.270888 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.270802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" event={"ID":"61674e54-e1a7-48cc-8eac-00b083ba49cc","Type":"ContainerDied","Data":"080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db"} Sep 11 19:46:52.270888 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.270870 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" Sep 11 19:46:52.270997 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.270872 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2" event={"ID":"61674e54-e1a7-48cc-8eac-00b083ba49cc","Type":"ContainerDied","Data":"0a51deb30a45d2c578c39f87a57585887f7f5620258ec7e0306f8b35a364b3d7"} Sep 11 19:46:52.280289 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.280269 2559 scope.go:117] "RemoveContainer" containerID="b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87" Sep 11 19:46:52.280654 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:46:52.280633 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87\": container with ID starting with b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87 not found: ID does not exist" containerID="b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87" Sep 11 19:46:52.280733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.280660 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87"} err="failed to get container status \"b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87\": rpc error: code = NotFound desc = could not find container \"b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87\": container with ID starting with b5eb395e703e7af6c622f38044271c03c3e288489026b4d251ddd29c20d15e87 not found: ID does not exist" Sep 11 19:46:52.280733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.280678 2559 scope.go:117] "RemoveContainer" containerID="080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db" Sep 11 19:46:52.283610 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.283590 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:46:52.291762 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.291738 2559 scope.go:117] "RemoveContainer" containerID="080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db" Sep 11 19:46:52.292122 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:46:52.292100 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db\": container with ID starting with 080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db not found: ID does not exist" containerID="080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db" Sep 11 19:46:52.292219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.292130 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db"} err="failed to get container status \"080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db\": rpc error: code = NotFound desc = could not find container \"080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db\": container with ID starting with 080073ed2742bae31b3f49a7a028d5faf0fad983f43169c2ac1d7d49b1ffe5db not found: ID does not exist" Sep 11 19:46:52.349293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.349256 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2"] Sep 11 19:46:52.357497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.357468 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-8901a783-7b9b5476d5-swtc2"] Sep 11 19:46:52.361155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.361127 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7lj\" (UniqueName: \"kubernetes.io/projected/61674e54-e1a7-48cc-8eac-00b083ba49cc-kube-api-access-zm7lj\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:52.361155 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.361156 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tts2q\" (UniqueName: \"kubernetes.io/projected/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-kube-api-access-tts2q\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:52.361364 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.361172 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa71e8b2-0626-4d76-87ac-32bbef8ba61c-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:46:52.594474 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.594423 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822"] Sep 11 19:46:52.611399 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:52.611372 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8901a783-76f4f6c4-b6822"] Sep 11 19:46:53.224846 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:53.224813 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03986d10-3a98-4e19-90b4-ac685d579f61" path="/var/lib/kubelet/pods/03986d10-3a98-4e19-90b4-ac685d579f61/volumes" Sep 11 19:46:53.225173 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:53.225160 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61674e54-e1a7-48cc-8eac-00b083ba49cc" path="/var/lib/kubelet/pods/61674e54-e1a7-48cc-8eac-00b083ba49cc/volumes" Sep 11 19:46:53.225491 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:53.225477 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa71e8b2-0626-4d76-87ac-32bbef8ba61c" path="/var/lib/kubelet/pods/fa71e8b2-0626-4d76-87ac-32bbef8ba61c/volumes" Sep 11 19:46:57.082402 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:57.082366 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:46:59.825549 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.825471 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w"] Sep 11 19:46:59.825928 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.825912 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa71e8b2-0626-4d76-87ac-32bbef8ba61c" containerName="payload-pre-processing" Sep 11 19:46:59.825973 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.825930 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa71e8b2-0626-4d76-87ac-32bbef8ba61c" containerName="payload-pre-processing" Sep 11 19:46:59.825973 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.825949 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03986d10-3a98-4e19-90b4-ac685d579f61" containerName="revoke-keys" Sep 11 19:46:59.825973 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.825955 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="03986d10-3a98-4e19-90b4-ac685d579f61" containerName="revoke-keys" Sep 11 19:46:59.825973 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.825966 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61674e54-e1a7-48cc-8eac-00b083ba49cc" containerName="payload-processing" Sep 11 19:46:59.825973 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.825972 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="61674e54-e1a7-48cc-8eac-00b083ba49cc" containerName="payload-processing" Sep 11 19:46:59.826124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.826044 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="61674e54-e1a7-48cc-8eac-00b083ba49cc" containerName="payload-processing" Sep 11 19:46:59.826124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.826052 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="03986d10-3a98-4e19-90b4-ac685d579f61" containerName="revoke-keys" Sep 11 19:46:59.826124 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.826060 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa71e8b2-0626-4d76-87ac-32bbef8ba61c" containerName="payload-pre-processing" Sep 11 19:46:59.830542 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.830521 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:46:59.837980 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.837958 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-0d707fde-dockercfg-msnxg\"" Sep 11 19:46:59.838069 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.837998 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-0d707fde\"" Sep 11 19:46:59.845244 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.845216 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w"] Sep 11 19:46:59.893832 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.893793 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w"] Sep 11 19:46:59.897360 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.897343 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:46:59.923394 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.923364 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w"] Sep 11 19:46:59.929283 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.929258 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199166e4-f466-443a-8709-2c16066c6a7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w\" (UID: \"199166e4-f466-443a-8709-2c16066c6a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:46:59.929400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.929300 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb45348a-69c0-4d59-86ae-e5e2a43eca09-plugins-config-volume\") pod \"payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w\" (UID: \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\") " pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:46:59.929400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.929381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4nlq\" (UniqueName: \"kubernetes.io/projected/eb45348a-69c0-4d59-86ae-e5e2a43eca09-kube-api-access-m4nlq\") pod \"payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w\" (UID: \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\") " pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:46:59.929497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:46:59.929406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2z9\" (UniqueName: \"kubernetes.io/projected/199166e4-f466-443a-8709-2c16066c6a7e-kube-api-access-sh2z9\") pod \"payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w\" (UID: \"199166e4-f466-443a-8709-2c16066c6a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:00.030496 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.030427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4nlq\" (UniqueName: \"kubernetes.io/projected/eb45348a-69c0-4d59-86ae-e5e2a43eca09-kube-api-access-m4nlq\") pod \"payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w\" (UID: \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\") " pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:00.030496 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.030504 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2z9\" (UniqueName: \"kubernetes.io/projected/199166e4-f466-443a-8709-2c16066c6a7e-kube-api-access-sh2z9\") pod \"payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w\" (UID: \"199166e4-f466-443a-8709-2c16066c6a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:00.030762 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.030580 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199166e4-f466-443a-8709-2c16066c6a7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w\" (UID: \"199166e4-f466-443a-8709-2c16066c6a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:00.030762 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.030610 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb45348a-69c0-4d59-86ae-e5e2a43eca09-plugins-config-volume\") pod \"payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w\" (UID: \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\") " pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:00.031265 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.031245 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199166e4-f466-443a-8709-2c16066c6a7e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w\" (UID: \"199166e4-f466-443a-8709-2c16066c6a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:00.031320 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.031286 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb45348a-69c0-4d59-86ae-e5e2a43eca09-plugins-config-volume\") pod \"payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w\" (UID: \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\") " pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:00.046172 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.046142 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4nlq\" (UniqueName: \"kubernetes.io/projected/eb45348a-69c0-4d59-86ae-e5e2a43eca09-kube-api-access-m4nlq\") pod \"payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w\" (UID: \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\") " pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:00.049973 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.049949 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2z9\" (UniqueName: \"kubernetes.io/projected/199166e4-f466-443a-8709-2c16066c6a7e-kube-api-access-sh2z9\") pod \"payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w\" (UID: \"199166e4-f466-443a-8709-2c16066c6a7e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:00.141614 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.141520 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:00.209302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.209019 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:00.439112 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.439088 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w"] Sep 11 19:47:00.439885 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:47:00.439862 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199166e4_f466_443a_8709_2c16066c6a7e.slice/crio-df5bfb56cdf81bbc2f6d8299074a6b12f3a1f8ded637798c1048570d708d1548 WatchSource:0}: Error finding container df5bfb56cdf81bbc2f6d8299074a6b12f3a1f8ded637798c1048570d708d1548: Status 404 returned error can't find the container with id df5bfb56cdf81bbc2f6d8299074a6b12f3a1f8ded637798c1048570d708d1548 Sep 11 19:47:00.512353 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:00.512324 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w"] Sep 11 19:47:00.513604 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:47:00.513577 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb45348a_69c0_4d59_86ae_e5e2a43eca09.slice/crio-45135d07e374a463bd1d8ac6eb42a4d7fb2fd2d1b78b16d233f1c761dc80ee0d WatchSource:0}: Error finding container 45135d07e374a463bd1d8ac6eb42a4d7fb2fd2d1b78b16d233f1c761dc80ee0d: Status 404 returned error can't find the container with id 45135d07e374a463bd1d8ac6eb42a4d7fb2fd2d1b78b16d233f1c761dc80ee0d Sep 11 19:47:01.338775 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:01.338737 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" event={"ID":"eb45348a-69c0-4d59-86ae-e5e2a43eca09","Type":"ContainerStarted","Data":"84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5"} Sep 11 19:47:01.339187 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:01.338928 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:01.339187 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:01.338961 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" event={"ID":"eb45348a-69c0-4d59-86ae-e5e2a43eca09","Type":"ContainerStarted","Data":"45135d07e374a463bd1d8ac6eb42a4d7fb2fd2d1b78b16d233f1c761dc80ee0d"} Sep 11 19:47:01.341191 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:01.341167 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" event={"ID":"199166e4-f466-443a-8709-2c16066c6a7e","Type":"ContainerStarted","Data":"a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e"} Sep 11 19:47:01.341338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:01.341211 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" event={"ID":"199166e4-f466-443a-8709-2c16066c6a7e","Type":"ContainerStarted","Data":"df5bfb56cdf81bbc2f6d8299074a6b12f3a1f8ded637798c1048570d708d1548"} Sep 11 19:47:01.341338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:01.341250 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:01.383323 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:01.383264 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" podStartSLOduration=2.383242234 podStartE2EDuration="2.383242234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:01.38305568 +0000 UTC m=+1254.655562881" watchObservedRunningTime="2026-09-11 19:47:01.383242234 +0000 UTC m=+1254.655749436" Sep 11 19:47:01.420044 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:01.419993 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" podStartSLOduration=2.419976803 podStartE2EDuration="2.419976803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:01.419642129 +0000 UTC m=+1254.692149330" watchObservedRunningTime="2026-09-11 19:47:01.419976803 +0000 UTC m=+1254.692484002" Sep 11 19:47:12.349948 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:12.349915 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:12.350338 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:12.349997 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:15.170276 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:15.170241 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:47:21.535393 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.535339 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq"] Sep 11 19:47:21.540378 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.540353 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" Sep 11 19:47:21.544651 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.544626 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:47:21.545585 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.545564 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:47:21.555611 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.555584 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq"] Sep 11 19:47:21.646809 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.646772 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e686a-710f-4321-9795-6cc3472a8dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq\" (UID: \"228e686a-710f-4321-9795-6cc3472a8dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" Sep 11 19:47:21.747686 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.747651 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e686a-710f-4321-9795-6cc3472a8dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq\" (UID: \"228e686a-710f-4321-9795-6cc3472a8dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" Sep 11 19:47:21.748345 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.748322 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e686a-710f-4321-9795-6cc3472a8dc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq\" (UID: \"228e686a-710f-4321-9795-6cc3472a8dc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" Sep 11 19:47:21.850490 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:21.850386 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" Sep 11 19:47:22.131698 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:22.131672 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq"] Sep 11 19:47:22.132679 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:47:22.132649 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod228e686a_710f_4321_9795_6cc3472a8dc2.slice/crio-feeac43a865e68dab417cc47c37ac1d4d83972c52e7e933512c4543f30a95361 WatchSource:0}: Error finding container feeac43a865e68dab417cc47c37ac1d4d83972c52e7e933512c4543f30a95361: Status 404 returned error can't find the container with id feeac43a865e68dab417cc47c37ac1d4d83972c52e7e933512c4543f30a95361 Sep 11 19:47:22.488454 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:22.488394 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" event={"ID":"228e686a-710f-4321-9795-6cc3472a8dc2","Type":"ContainerStarted","Data":"f07a0a8284e81fb9552f184a10ed98bb37731e994c0c962509976c7783fe4b04"} Sep 11 19:47:22.488611 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:22.488515 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" event={"ID":"228e686a-710f-4321-9795-6cc3472a8dc2","Type":"ContainerStarted","Data":"feeac43a865e68dab417cc47c37ac1d4d83972c52e7e933512c4543f30a95361"} Sep 11 19:47:22.514067 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:22.514014 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" podStartSLOduration=1.5139952779999999 podStartE2EDuration="1.513995278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:47:22.512116673 +0000 UTC m=+1275.784623875" watchObservedRunningTime="2026-09-11 19:47:22.513995278 +0000 UTC m=+1275.786502478" Sep 11 19:47:23.496693 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:23.496659 2559 generic.go:356] "Generic (PLEG): container finished" podID="228e686a-710f-4321-9795-6cc3472a8dc2" containerID="f07a0a8284e81fb9552f184a10ed98bb37731e994c0c962509976c7783fe4b04" exitCode=0 Sep 11 19:47:23.497152 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:23.496745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" event={"ID":"228e686a-710f-4321-9795-6cc3472a8dc2","Type":"ContainerDied","Data":"f07a0a8284e81fb9552f184a10ed98bb37731e994c0c962509976c7783fe4b04"} Sep 11 19:47:24.647502 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:24.647476 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" Sep 11 19:47:24.777499 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:24.777394 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e686a-710f-4321-9795-6cc3472a8dc2-maas-api-service-ca\") pod \"228e686a-710f-4321-9795-6cc3472a8dc2\" (UID: \"228e686a-710f-4321-9795-6cc3472a8dc2\") " Sep 11 19:47:24.777844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:24.777816 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/228e686a-710f-4321-9795-6cc3472a8dc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "228e686a-710f-4321-9795-6cc3472a8dc2" (UID: "228e686a-710f-4321-9795-6cc3472a8dc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:47:24.879139 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:24.879102 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/228e686a-710f-4321-9795-6cc3472a8dc2-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:47:25.513356 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:25.513322 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" event={"ID":"228e686a-710f-4321-9795-6cc3472a8dc2","Type":"ContainerDied","Data":"feeac43a865e68dab417cc47c37ac1d4d83972c52e7e933512c4543f30a95361"} Sep 11 19:47:25.513555 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:25.513367 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq" Sep 11 19:47:25.513555 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:25.513377 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="feeac43a865e68dab417cc47c37ac1d4d83972c52e7e933512c4543f30a95361" Sep 11 19:47:29.134756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.134722 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w"] Sep 11 19:47:29.135175 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.134925 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" podUID="eb45348a-69c0-4d59-86ae-e5e2a43eca09" containerName="payload-processing" containerID="cri-o://84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5" gracePeriod=30 Sep 11 19:47:29.142121 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.142091 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w"] Sep 11 19:47:29.142360 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.142334 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" podUID="199166e4-f466-443a-8709-2c16066c6a7e" containerName="payload-pre-processing" containerID="cri-o://a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e" gracePeriod=30 Sep 11 19:47:29.396256 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.396189 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:29.413806 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.413780 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq"] Sep 11 19:47:29.419597 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.419567 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb45348a-69c0-4d59-86ae-e5e2a43eca09-plugins-config-volume\") pod \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\" (UID: \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\") " Sep 11 19:47:29.419787 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.419618 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4nlq\" (UniqueName: \"kubernetes.io/projected/eb45348a-69c0-4d59-86ae-e5e2a43eca09-kube-api-access-m4nlq\") pod \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\" (UID: \"eb45348a-69c0-4d59-86ae-e5e2a43eca09\") " Sep 11 19:47:29.420142 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.420058 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb45348a-69c0-4d59-86ae-e5e2a43eca09-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "eb45348a-69c0-4d59-86ae-e5e2a43eca09" (UID: "eb45348a-69c0-4d59-86ae-e5e2a43eca09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:47:29.422930 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.422869 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb45348a-69c0-4d59-86ae-e5e2a43eca09-kube-api-access-m4nlq" (OuterVolumeSpecName: "kube-api-access-m4nlq") pod "eb45348a-69c0-4d59-86ae-e5e2a43eca09" (UID: "eb45348a-69c0-4d59-86ae-e5e2a43eca09"). InnerVolumeSpecName "kube-api-access-m4nlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:47:29.424886 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.424862 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0d707fde-4vsfq"] Sep 11 19:47:29.433704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.433683 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:29.520478 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.520427 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199166e4-f466-443a-8709-2c16066c6a7e-plugins-config-volume\") pod \"199166e4-f466-443a-8709-2c16066c6a7e\" (UID: \"199166e4-f466-443a-8709-2c16066c6a7e\") " Sep 11 19:47:29.520478 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.520484 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sh2z9\" (UniqueName: \"kubernetes.io/projected/199166e4-f466-443a-8709-2c16066c6a7e-kube-api-access-sh2z9\") pod \"199166e4-f466-443a-8709-2c16066c6a7e\" (UID: \"199166e4-f466-443a-8709-2c16066c6a7e\") " Sep 11 19:47:29.520732 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.520721 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb45348a-69c0-4d59-86ae-e5e2a43eca09-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:47:29.520772 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.520734 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4nlq\" (UniqueName: \"kubernetes.io/projected/eb45348a-69c0-4d59-86ae-e5e2a43eca09-kube-api-access-m4nlq\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:47:29.520900 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.520863 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/199166e4-f466-443a-8709-2c16066c6a7e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "199166e4-f466-443a-8709-2c16066c6a7e" (UID: "199166e4-f466-443a-8709-2c16066c6a7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:47:29.523063 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.523037 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199166e4-f466-443a-8709-2c16066c6a7e-kube-api-access-sh2z9" (OuterVolumeSpecName: "kube-api-access-sh2z9") pod "199166e4-f466-443a-8709-2c16066c6a7e" (UID: "199166e4-f466-443a-8709-2c16066c6a7e"). InnerVolumeSpecName "kube-api-access-sh2z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:47:29.542108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.542073 2559 generic.go:356] "Generic (PLEG): container finished" podID="eb45348a-69c0-4d59-86ae-e5e2a43eca09" containerID="84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5" exitCode=0 Sep 11 19:47:29.542295 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.542129 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" Sep 11 19:47:29.542295 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.542161 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" event={"ID":"eb45348a-69c0-4d59-86ae-e5e2a43eca09","Type":"ContainerDied","Data":"84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5"} Sep 11 19:47:29.542295 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.542225 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w" event={"ID":"eb45348a-69c0-4d59-86ae-e5e2a43eca09","Type":"ContainerDied","Data":"45135d07e374a463bd1d8ac6eb42a4d7fb2fd2d1b78b16d233f1c761dc80ee0d"} Sep 11 19:47:29.542295 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.542257 2559 scope.go:117] "RemoveContainer" containerID="84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5" Sep 11 19:47:29.543976 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.543952 2559 generic.go:356] "Generic (PLEG): container finished" podID="199166e4-f466-443a-8709-2c16066c6a7e" containerID="a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e" exitCode=0 Sep 11 19:47:29.544108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.543996 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" Sep 11 19:47:29.544108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.544026 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" event={"ID":"199166e4-f466-443a-8709-2c16066c6a7e","Type":"ContainerDied","Data":"a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e"} Sep 11 19:47:29.544108 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.544066 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w" event={"ID":"199166e4-f466-443a-8709-2c16066c6a7e","Type":"ContainerDied","Data":"df5bfb56cdf81bbc2f6d8299074a6b12f3a1f8ded637798c1048570d708d1548"} Sep 11 19:47:29.555341 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.554981 2559 scope.go:117] "RemoveContainer" containerID="84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5" Sep 11 19:47:29.555341 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:47:29.555301 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5\": container with ID starting with 84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5 not found: ID does not exist" containerID="84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5" Sep 11 19:47:29.555341 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.555326 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5"} err="failed to get container status \"84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5\": rpc error: code = NotFound desc = could not find container \"84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5\": container with ID starting with 84185aad9f2d96d9da5ac19e2f4c7be70399ca209d2bf9aeb895321ece35a9d5 not found: ID does not exist" Sep 11 19:47:29.555549 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.555344 2559 scope.go:117] "RemoveContainer" containerID="a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e" Sep 11 19:47:29.564510 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.564487 2559 scope.go:117] "RemoveContainer" containerID="a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e" Sep 11 19:47:29.564791 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:47:29.564773 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e\": container with ID starting with a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e not found: ID does not exist" containerID="a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e" Sep 11 19:47:29.564843 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.564800 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e"} err="failed to get container status \"a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e\": rpc error: code = NotFound desc = could not find container \"a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e\": container with ID starting with a6ef923e6ffbf1850dfae34b4f941d9ae30894efab88f4091bcc06ad56b1459e not found: ID does not exist" Sep 11 19:47:29.606117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.606089 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w"] Sep 11 19:47:29.620751 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.620727 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-0d707fde-756b9ccd57-5fn5w"] Sep 11 19:47:29.621142 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.621128 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/199166e4-f466-443a-8709-2c16066c6a7e-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:47:29.621192 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.621145 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh2z9\" (UniqueName: \"kubernetes.io/projected/199166e4-f466-443a-8709-2c16066c6a7e-kube-api-access-sh2z9\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:47:29.657753 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.657665 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w"] Sep 11 19:47:29.671290 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:29.671261 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-0d707fde-54b75c55f8-4t22w"] Sep 11 19:47:30.708216 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:30.708183 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:47:31.224814 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:31.224776 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199166e4-f466-443a-8709-2c16066c6a7e" path="/var/lib/kubelet/pods/199166e4-f466-443a-8709-2c16066c6a7e/volumes" Sep 11 19:47:31.225168 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:31.225152 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="228e686a-710f-4321-9795-6cc3472a8dc2" path="/var/lib/kubelet/pods/228e686a-710f-4321-9795-6cc3472a8dc2/volumes" Sep 11 19:47:31.225484 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:31.225472 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb45348a-69c0-4d59-86ae-e5e2a43eca09" path="/var/lib/kubelet/pods/eb45348a-69c0-4d59-86ae-e5e2a43eca09/volumes" Sep 11 19:47:45.397486 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:45.397427 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:47:49.047267 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.047219 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc"] Sep 11 19:47:49.048257 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048230 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb45348a-69c0-4d59-86ae-e5e2a43eca09" containerName="payload-processing" Sep 11 19:47:49.048535 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048507 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb45348a-69c0-4d59-86ae-e5e2a43eca09" containerName="payload-processing" Sep 11 19:47:49.048638 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048549 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="228e686a-710f-4321-9795-6cc3472a8dc2" containerName="revoke-keys" Sep 11 19:47:49.048638 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048560 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="228e686a-710f-4321-9795-6cc3472a8dc2" containerName="revoke-keys" Sep 11 19:47:49.048638 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048605 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199166e4-f466-443a-8709-2c16066c6a7e" containerName="payload-pre-processing" Sep 11 19:47:49.048638 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048613 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="199166e4-f466-443a-8709-2c16066c6a7e" containerName="payload-pre-processing" Sep 11 19:47:49.048890 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048873 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="228e686a-710f-4321-9795-6cc3472a8dc2" containerName="revoke-keys" Sep 11 19:47:49.048969 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048894 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb45348a-69c0-4d59-86ae-e5e2a43eca09" containerName="payload-processing" Sep 11 19:47:49.048969 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.048918 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="199166e4-f466-443a-8709-2c16066c6a7e" containerName="payload-pre-processing" Sep 11 19:47:49.053303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.053275 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" Sep 11 19:47:49.056592 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.056567 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:47:49.056951 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.056926 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:47:49.058909 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.058887 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc"] Sep 11 19:47:49.116575 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.116534 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ea6faf-548f-4a95-9655-524910963c42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6bf598-f84pc\" (UID: \"45ea6faf-548f-4a95-9655-524910963c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" Sep 11 19:47:49.217454 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.217402 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ea6faf-548f-4a95-9655-524910963c42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6bf598-f84pc\" (UID: \"45ea6faf-548f-4a95-9655-524910963c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" Sep 11 19:47:49.218047 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.218025 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ea6faf-548f-4a95-9655-524910963c42-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-6bf598-f84pc\" (UID: \"45ea6faf-548f-4a95-9655-524910963c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" Sep 11 19:47:49.365839 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.365755 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" Sep 11 19:47:49.650008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.649979 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc"] Sep 11 19:47:49.650885 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:47:49.650859 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ea6faf_548f_4a95_9655_524910963c42.slice/crio-15db17300c1ef763df22f0b4adedfbb270987eb5e32138e8c08063ace6749ed1 WatchSource:0}: Error finding container 15db17300c1ef763df22f0b4adedfbb270987eb5e32138e8c08063ace6749ed1: Status 404 returned error can't find the container with id 15db17300c1ef763df22f0b4adedfbb270987eb5e32138e8c08063ace6749ed1 Sep 11 19:47:49.678069 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:49.678042 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" event={"ID":"45ea6faf-548f-4a95-9655-524910963c42","Type":"ContainerStarted","Data":"15db17300c1ef763df22f0b4adedfbb270987eb5e32138e8c08063ace6749ed1"} Sep 11 19:47:50.686545 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:50.686429 2559 generic.go:356] "Generic (PLEG): container finished" podID="45ea6faf-548f-4a95-9655-524910963c42" containerID="19849a6b8ed86e970025c22bcd8591e4397cbf6a94f181dc5c73797e66bef5b8" exitCode=0 Sep 11 19:47:50.686545 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:50.686463 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" event={"ID":"45ea6faf-548f-4a95-9655-524910963c42","Type":"ContainerDied","Data":"19849a6b8ed86e970025c22bcd8591e4397cbf6a94f181dc5c73797e66bef5b8"} Sep 11 19:47:51.824259 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:51.824235 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" Sep 11 19:47:51.942899 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:51.942866 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ea6faf-548f-4a95-9655-524910963c42-maas-api-service-ca\") pod \"45ea6faf-548f-4a95-9655-524910963c42\" (UID: \"45ea6faf-548f-4a95-9655-524910963c42\") " Sep 11 19:47:51.943273 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:51.943216 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ea6faf-548f-4a95-9655-524910963c42-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45ea6faf-548f-4a95-9655-524910963c42" (UID: "45ea6faf-548f-4a95-9655-524910963c42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:47:52.044280 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:52.044237 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45ea6faf-548f-4a95-9655-524910963c42-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:47:52.705335 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:52.705300 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" event={"ID":"45ea6faf-548f-4a95-9655-524910963c42","Type":"ContainerDied","Data":"15db17300c1ef763df22f0b4adedfbb270987eb5e32138e8c08063ace6749ed1"} Sep 11 19:47:52.705335 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:52.705323 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc" Sep 11 19:47:52.705638 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:52.705360 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15db17300c1ef763df22f0b4adedfbb270987eb5e32138e8c08063ace6749ed1" Sep 11 19:47:59.598429 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.598397 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj"] Sep 11 19:47:59.599050 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.598660 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" podUID="45ca6c7f-0688-4d27-a70c-77df94606890" containerName="payload-processing" containerID="cri-o://14b655da7bdf24e53d04f548f577d9ab929773fc85908b45f9096b21d128721a" gracePeriod=30 Sep 11 19:47:59.607886 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.607861 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk"] Sep 11 19:47:59.608303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.608274 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" podUID="67f04ff2-d7ee-45a1-a3d3-f435213a4a89" containerName="payload-pre-processing" containerID="cri-o://8562547ae49616014117cbd0640a27ea1ce7571d565bc8d9ced634a1fb144bce" gracePeriod=30 Sep 11 19:47:59.757823 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.757788 2559 generic.go:356] "Generic (PLEG): container finished" podID="45ca6c7f-0688-4d27-a70c-77df94606890" containerID="14b655da7bdf24e53d04f548f577d9ab929773fc85908b45f9096b21d128721a" exitCode=0 Sep 11 19:47:59.758008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.757866 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" event={"ID":"45ca6c7f-0688-4d27-a70c-77df94606890","Type":"ContainerDied","Data":"14b655da7bdf24e53d04f548f577d9ab929773fc85908b45f9096b21d128721a"} Sep 11 19:47:59.759607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.759583 2559 generic.go:356] "Generic (PLEG): container finished" podID="67f04ff2-d7ee-45a1-a3d3-f435213a4a89" containerID="8562547ae49616014117cbd0640a27ea1ce7571d565bc8d9ced634a1fb144bce" exitCode=0 Sep 11 19:47:59.759705 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.759613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" event={"ID":"67f04ff2-d7ee-45a1-a3d3-f435213a4a89","Type":"ContainerDied","Data":"8562547ae49616014117cbd0640a27ea1ce7571d565bc8d9ced634a1fb144bce"} Sep 11 19:47:59.769112 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.769083 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc"] Sep 11 19:47:59.772411 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.772377 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-6bf598-f84pc"] Sep 11 19:47:59.873329 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.873302 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:47:59.894093 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:47:59.894067 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:48:00.025922 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.025885 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca6c7f-0688-4d27-a70c-77df94606890-plugins-config-volume\") pod \"45ca6c7f-0688-4d27-a70c-77df94606890\" (UID: \"45ca6c7f-0688-4d27-a70c-77df94606890\") " Sep 11 19:48:00.026118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.025930 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-plugins-config-volume\") pod \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\" (UID: \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\") " Sep 11 19:48:00.026118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.025968 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv8jj\" (UniqueName: \"kubernetes.io/projected/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-kube-api-access-cv8jj\") pod \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\" (UID: \"67f04ff2-d7ee-45a1-a3d3-f435213a4a89\") " Sep 11 19:48:00.026118 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.025996 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk8vx\" (UniqueName: \"kubernetes.io/projected/45ca6c7f-0688-4d27-a70c-77df94606890-kube-api-access-fk8vx\") pod \"45ca6c7f-0688-4d27-a70c-77df94606890\" (UID: \"45ca6c7f-0688-4d27-a70c-77df94606890\") " Sep 11 19:48:00.026395 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.026369 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67f04ff2-d7ee-45a1-a3d3-f435213a4a89" (UID: "67f04ff2-d7ee-45a1-a3d3-f435213a4a89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:48:00.026496 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.026365 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ca6c7f-0688-4d27-a70c-77df94606890-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45ca6c7f-0688-4d27-a70c-77df94606890" (UID: "45ca6c7f-0688-4d27-a70c-77df94606890"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:48:00.028268 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.028248 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ca6c7f-0688-4d27-a70c-77df94606890-kube-api-access-fk8vx" (OuterVolumeSpecName: "kube-api-access-fk8vx") pod "45ca6c7f-0688-4d27-a70c-77df94606890" (UID: "45ca6c7f-0688-4d27-a70c-77df94606890"). InnerVolumeSpecName "kube-api-access-fk8vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:48:00.028334 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.028267 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-kube-api-access-cv8jj" (OuterVolumeSpecName: "kube-api-access-cv8jj") pod "67f04ff2-d7ee-45a1-a3d3-f435213a4a89" (UID: "67f04ff2-d7ee-45a1-a3d3-f435213a4a89"). InnerVolumeSpecName "kube-api-access-cv8jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:48:00.127213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.127120 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ca6c7f-0688-4d27-a70c-77df94606890-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:48:00.127213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.127152 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:48:00.127213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.127163 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv8jj\" (UniqueName: \"kubernetes.io/projected/67f04ff2-d7ee-45a1-a3d3-f435213a4a89-kube-api-access-cv8jj\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:48:00.127213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.127171 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk8vx\" (UniqueName: \"kubernetes.io/projected/45ca6c7f-0688-4d27-a70c-77df94606890-kube-api-access-fk8vx\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:48:00.768565 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.768531 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" Sep 11 19:48:00.768565 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.768545 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj" event={"ID":"45ca6c7f-0688-4d27-a70c-77df94606890","Type":"ContainerDied","Data":"b7cfc0b905f53bf971c10801b9171dc4fa1ce9858032506600a37596b3fa046e"} Sep 11 19:48:00.769074 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.768612 2559 scope.go:117] "RemoveContainer" containerID="14b655da7bdf24e53d04f548f577d9ab929773fc85908b45f9096b21d128721a" Sep 11 19:48:00.770143 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.770117 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" Sep 11 19:48:00.770454 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.770126 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk" event={"ID":"67f04ff2-d7ee-45a1-a3d3-f435213a4a89","Type":"ContainerDied","Data":"6e4b99b7d221200465f3d2ff07b8d89b899575d82591723253521befc17e1f23"} Sep 11 19:48:00.779969 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.779941 2559 scope.go:117] "RemoveContainer" containerID="8562547ae49616014117cbd0640a27ea1ce7571d565bc8d9ced634a1fb144bce" Sep 11 19:48:00.793873 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.793842 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj"] Sep 11 19:48:00.797585 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.797562 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-6bf598-855464d6f7-88jmj"] Sep 11 19:48:00.807705 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.807674 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk"] Sep 11 19:48:00.813236 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:00.813215 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-6bf598-7dc668bbf4-29bwk"] Sep 11 19:48:01.224203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:01.224172 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ca6c7f-0688-4d27-a70c-77df94606890" path="/var/lib/kubelet/pods/45ca6c7f-0688-4d27-a70c-77df94606890/volumes" Sep 11 19:48:01.224544 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:01.224530 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ea6faf-548f-4a95-9655-524910963c42" path="/var/lib/kubelet/pods/45ea6faf-548f-4a95-9655-524910963c42/volumes" Sep 11 19:48:01.224844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:01.224833 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67f04ff2-d7ee-45a1-a3d3-f435213a4a89" path="/var/lib/kubelet/pods/67f04ff2-d7ee-45a1-a3d3-f435213a4a89/volumes" Sep 11 19:48:07.095332 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:07.095287 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:48:38.463331 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:38.463234 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:48:49.725954 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.725914 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f"] Sep 11 19:48:49.726423 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726316 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67f04ff2-d7ee-45a1-a3d3-f435213a4a89" containerName="payload-pre-processing" Sep 11 19:48:49.726423 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726326 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="67f04ff2-d7ee-45a1-a3d3-f435213a4a89" containerName="payload-pre-processing" Sep 11 19:48:49.726423 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726347 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ca6c7f-0688-4d27-a70c-77df94606890" containerName="payload-processing" Sep 11 19:48:49.726423 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726352 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ca6c7f-0688-4d27-a70c-77df94606890" containerName="payload-processing" Sep 11 19:48:49.726423 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726359 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ea6faf-548f-4a95-9655-524910963c42" containerName="revoke-keys" Sep 11 19:48:49.726423 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726365 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ea6faf-548f-4a95-9655-524910963c42" containerName="revoke-keys" Sep 11 19:48:49.726423 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726425 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ea6faf-548f-4a95-9655-524910963c42" containerName="revoke-keys" Sep 11 19:48:49.726699 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726451 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="67f04ff2-d7ee-45a1-a3d3-f435213a4a89" containerName="payload-pre-processing" Sep 11 19:48:49.726699 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.726458 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ca6c7f-0688-4d27-a70c-77df94606890" containerName="payload-processing" Sep 11 19:48:49.729656 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.729642 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:49.733737 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.733714 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-a29559-dockercfg-fxzcr\"" Sep 11 19:48:49.733863 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.733816 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-a29559\"" Sep 11 19:48:49.743757 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.743734 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f"] Sep 11 19:48:49.782738 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.782708 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7"] Sep 11 19:48:49.786235 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.786217 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:49.814843 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.814812 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7"] Sep 11 19:48:49.898038 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.898003 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f\" (UID: \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:49.898216 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.898049 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5hv\" (UniqueName: \"kubernetes.io/projected/72129686-8a3d-42cc-b711-4718199eaed7-kube-api-access-8p5hv\") pod \"payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7\" (UID: \"72129686-8a3d-42cc-b711-4718199eaed7\") " pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:49.898216 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.898140 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wg48\" (UniqueName: \"kubernetes.io/projected/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-kube-api-access-5wg48\") pod \"payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f\" (UID: \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:49.898216 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.898172 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72129686-8a3d-42cc-b711-4718199eaed7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7\" (UID: \"72129686-8a3d-42cc-b711-4718199eaed7\") " pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:49.998884 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.998839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wg48\" (UniqueName: \"kubernetes.io/projected/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-kube-api-access-5wg48\") pod \"payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f\" (UID: \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:49.999092 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.998898 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72129686-8a3d-42cc-b711-4718199eaed7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7\" (UID: \"72129686-8a3d-42cc-b711-4718199eaed7\") " pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:49.999092 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.999015 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f\" (UID: \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:49.999092 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.999055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5hv\" (UniqueName: \"kubernetes.io/projected/72129686-8a3d-42cc-b711-4718199eaed7-kube-api-access-8p5hv\") pod \"payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7\" (UID: \"72129686-8a3d-42cc-b711-4718199eaed7\") " pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:49.999884 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:49.999837 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72129686-8a3d-42cc-b711-4718199eaed7-plugins-config-volume\") pod \"payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7\" (UID: \"72129686-8a3d-42cc-b711-4718199eaed7\") " pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:50.000080 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:50.000049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f\" (UID: \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:50.016566 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:50.016537 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wg48\" (UniqueName: \"kubernetes.io/projected/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-kube-api-access-5wg48\") pod \"payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f\" (UID: \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:50.016715 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:50.016633 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5hv\" (UniqueName: \"kubernetes.io/projected/72129686-8a3d-42cc-b711-4718199eaed7-kube-api-access-8p5hv\") pod \"payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7\" (UID: \"72129686-8a3d-42cc-b711-4718199eaed7\") " pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:50.040497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:50.040466 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:50.096145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:50.095950 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:50.330717 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:50.330690 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f"] Sep 11 19:48:50.332226 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:48:50.332195 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48fdf415_2b4e_4b69_9ff6_87e234ba08cf.slice/crio-f45b0e793c95b52695c89ec6cf9de43d6baf2dec1d97bf4f3bc0cb7d45d20c54 WatchSource:0}: Error finding container f45b0e793c95b52695c89ec6cf9de43d6baf2dec1d97bf4f3bc0cb7d45d20c54: Status 404 returned error can't find the container with id f45b0e793c95b52695c89ec6cf9de43d6baf2dec1d97bf4f3bc0cb7d45d20c54 Sep 11 19:48:50.385390 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:50.385364 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7"] Sep 11 19:48:50.386633 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:48:50.386607 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72129686_8a3d_42cc_b711_4718199eaed7.slice/crio-1e0ea7559b4a44ff6acfe2bbfaaf6da567882b53d9210a8f38af1922ff6a87be WatchSource:0}: Error finding container 1e0ea7559b4a44ff6acfe2bbfaaf6da567882b53d9210a8f38af1922ff6a87be: Status 404 returned error can't find the container with id 1e0ea7559b4a44ff6acfe2bbfaaf6da567882b53d9210a8f38af1922ff6a87be Sep 11 19:48:51.104311 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:51.104262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" event={"ID":"72129686-8a3d-42cc-b711-4718199eaed7","Type":"ContainerStarted","Data":"aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522"} Sep 11 19:48:51.104838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:51.104335 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" event={"ID":"72129686-8a3d-42cc-b711-4718199eaed7","Type":"ContainerStarted","Data":"1e0ea7559b4a44ff6acfe2bbfaaf6da567882b53d9210a8f38af1922ff6a87be"} Sep 11 19:48:51.104838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:51.104516 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:48:51.107087 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:51.107049 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" event={"ID":"48fdf415-2b4e-4b69-9ff6-87e234ba08cf","Type":"ContainerStarted","Data":"92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a"} Sep 11 19:48:51.107237 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:51.107116 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" event={"ID":"48fdf415-2b4e-4b69-9ff6-87e234ba08cf","Type":"ContainerStarted","Data":"f45b0e793c95b52695c89ec6cf9de43d6baf2dec1d97bf4f3bc0cb7d45d20c54"} Sep 11 19:48:51.107298 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:51.107246 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:48:51.125706 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:51.125643 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" podStartSLOduration=2.12562286 podStartE2EDuration="2.12562286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:48:51.123538451 +0000 UTC m=+1364.396045652" watchObservedRunningTime="2026-09-11 19:48:51.12562286 +0000 UTC m=+1364.398130061" Sep 11 19:48:51.145943 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:51.145880 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" podStartSLOduration=2.145858724 podStartE2EDuration="2.145858724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:48:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:48:51.144172422 +0000 UTC m=+1364.416679620" watchObservedRunningTime="2026-09-11 19:48:51.145858724 +0000 UTC m=+1364.418365925" Sep 11 19:48:58.973759 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:48:58.973707 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:49:02.115943 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:02.115906 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:49:02.116359 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:02.116269 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:49:16.989657 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:16.989620 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v"] Sep 11 19:49:16.993400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:16.993381 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:16.996081 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:16.996059 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-02df91be-dockercfg-5hd6h\"" Sep 11 19:49:16.996207 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:16.996064 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-02df91be\"" Sep 11 19:49:17.005062 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.005035 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr"] Sep 11 19:49:17.010985 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.010960 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:17.017778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.017748 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v"] Sep 11 19:49:17.021486 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.021461 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr"] Sep 11 19:49:17.062698 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.062660 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfgwd\" (UniqueName: \"kubernetes.io/projected/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-kube-api-access-hfgwd\") pod \"payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v\" (UID: \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:17.062865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.062794 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v\" (UID: \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:17.163525 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.163474 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v\" (UID: \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:17.163704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.163609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfgwd\" (UniqueName: \"kubernetes.io/projected/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-kube-api-access-hfgwd\") pod \"payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v\" (UID: \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:17.163704 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.163664 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-plugins-config-volume\") pod \"payload-processing-e2e-mt-02df91be-59644dccf7-8lptr\" (UID: \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\") " pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:17.163785 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.163716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4qx\" (UniqueName: \"kubernetes.io/projected/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-kube-api-access-gp4qx\") pod \"payload-processing-e2e-mt-02df91be-59644dccf7-8lptr\" (UID: \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\") " pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:17.164232 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.164212 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v\" (UID: \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:17.171802 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.171775 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfgwd\" (UniqueName: \"kubernetes.io/projected/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-kube-api-access-hfgwd\") pod \"payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v\" (UID: \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:17.264416 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.264334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-plugins-config-volume\") pod \"payload-processing-e2e-mt-02df91be-59644dccf7-8lptr\" (UID: \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\") " pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:17.264416 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.264377 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4qx\" (UniqueName: \"kubernetes.io/projected/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-kube-api-access-gp4qx\") pod \"payload-processing-e2e-mt-02df91be-59644dccf7-8lptr\" (UID: \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\") " pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:17.265024 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.265004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-plugins-config-volume\") pod \"payload-processing-e2e-mt-02df91be-59644dccf7-8lptr\" (UID: \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\") " pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:17.272656 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.272634 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4qx\" (UniqueName: \"kubernetes.io/projected/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-kube-api-access-gp4qx\") pod \"payload-processing-e2e-mt-02df91be-59644dccf7-8lptr\" (UID: \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\") " pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:17.318329 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.318293 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:17.328269 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.328244 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:17.561803 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.561722 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:49:17.604886 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:49:17.604858 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e32aba7_5a18_4429_b0c9_09bab80aa4f0.slice/crio-f5dcad8a8cde6fffa7dea4b222697c6035a28a880098b10249054d5f364c1b75 WatchSource:0}: Error finding container f5dcad8a8cde6fffa7dea4b222697c6035a28a880098b10249054d5f364c1b75: Status 404 returned error can't find the container with id f5dcad8a8cde6fffa7dea4b222697c6035a28a880098b10249054d5f364c1b75 Sep 11 19:49:17.608865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.608842 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v"] Sep 11 19:49:17.620654 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:17.620629 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr"] Sep 11 19:49:17.622145 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:49:17.622112 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c88f9e9_cb87_486f_a4eb_9dd5b07e2184.slice/crio-d31d53843b58c4f169b9e8f169529a0219d46efa4c0dd3fc7e62ba6207ea6355 WatchSource:0}: Error finding container d31d53843b58c4f169b9e8f169529a0219d46efa4c0dd3fc7e62ba6207ea6355: Status 404 returned error can't find the container with id d31d53843b58c4f169b9e8f169529a0219d46efa4c0dd3fc7e62ba6207ea6355 Sep 11 19:49:18.294195 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:18.294160 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" event={"ID":"9e32aba7-5a18-4429-b0c9-09bab80aa4f0","Type":"ContainerStarted","Data":"99d2e60eaf2dd28ffae2c7d452aa68a9a89885812d33ff56615eca75bd3c5fdc"} Sep 11 19:49:18.294689 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:18.294224 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:18.294689 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:18.294249 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" event={"ID":"9e32aba7-5a18-4429-b0c9-09bab80aa4f0","Type":"ContainerStarted","Data":"f5dcad8a8cde6fffa7dea4b222697c6035a28a880098b10249054d5f364c1b75"} Sep 11 19:49:18.296069 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:18.296046 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" event={"ID":"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184","Type":"ContainerStarted","Data":"2f413210e12da4c9218cf0e996d08ff3759885877ffb4b6e259edf4785e37a87"} Sep 11 19:49:18.296177 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:18.296090 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" event={"ID":"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184","Type":"ContainerStarted","Data":"d31d53843b58c4f169b9e8f169529a0219d46efa4c0dd3fc7e62ba6207ea6355"} Sep 11 19:49:18.296177 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:18.296113 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:18.322254 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:18.322200 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" podStartSLOduration=2.322184616 podStartE2EDuration="2.322184616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:49:18.320410459 +0000 UTC m=+1391.592917658" watchObservedRunningTime="2026-09-11 19:49:18.322184616 +0000 UTC m=+1391.594691826" Sep 11 19:49:18.337123 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:18.337062 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" podStartSLOduration=2.337041819 podStartE2EDuration="2.337041819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:49:18.336511297 +0000 UTC m=+1391.609018499" watchObservedRunningTime="2026-09-11 19:49:18.337041819 +0000 UTC m=+1391.609549019" Sep 11 19:49:29.306095 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:29.306056 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:29.306707 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:29.306188 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:42.067151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:42.067112 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:49:48.691348 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:48.691305 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7"] Sep 11 19:49:48.695152 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:48.695134 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" Sep 11 19:49:48.697679 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:48.697659 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:49:48.697785 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:48.697698 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:49:48.702936 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:48.702908 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7"] Sep 11 19:49:48.764067 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:48.764025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c23e4bb-1abd-4ff7-a1af-e71e32c579a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02df91be-z6dk7\" (UID: \"4c23e4bb-1abd-4ff7-a1af-e71e32c579a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" Sep 11 19:49:48.865058 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:48.865027 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c23e4bb-1abd-4ff7-a1af-e71e32c579a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02df91be-z6dk7\" (UID: \"4c23e4bb-1abd-4ff7-a1af-e71e32c579a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" Sep 11 19:49:48.865715 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:48.865694 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c23e4bb-1abd-4ff7-a1af-e71e32c579a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-02df91be-z6dk7\" (UID: \"4c23e4bb-1abd-4ff7-a1af-e71e32c579a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" Sep 11 19:49:49.010792 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:49.010752 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" Sep 11 19:49:49.289077 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:49.289048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7"] Sep 11 19:49:49.290126 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:49:49.290100 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c23e4bb_1abd_4ff7_a1af_e71e32c579a1.slice/crio-71929a04f8115362e2901f3a0aed1a416330a39a26bba5ea9b520b22ff347149 WatchSource:0}: Error finding container 71929a04f8115362e2901f3a0aed1a416330a39a26bba5ea9b520b22ff347149: Status 404 returned error can't find the container with id 71929a04f8115362e2901f3a0aed1a416330a39a26bba5ea9b520b22ff347149 Sep 11 19:49:49.498792 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:49.498751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" event={"ID":"4c23e4bb-1abd-4ff7-a1af-e71e32c579a1","Type":"ContainerStarted","Data":"71929a04f8115362e2901f3a0aed1a416330a39a26bba5ea9b520b22ff347149"} Sep 11 19:49:50.509270 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:50.509237 2559 generic.go:356] "Generic (PLEG): container finished" podID="4c23e4bb-1abd-4ff7-a1af-e71e32c579a1" containerID="718bbd6d9802c5125d91970bd02bd4dcae4b26ea70ed338f4fc4ebcb9d9ac060" exitCode=0 Sep 11 19:49:50.509685 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:50.509328 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" event={"ID":"4c23e4bb-1abd-4ff7-a1af-e71e32c579a1","Type":"ContainerDied","Data":"718bbd6d9802c5125d91970bd02bd4dcae4b26ea70ed338f4fc4ebcb9d9ac060"} Sep 11 19:49:51.659297 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:51.659272 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" Sep 11 19:49:51.795012 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:51.794929 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c23e4bb-1abd-4ff7-a1af-e71e32c579a1-maas-api-service-ca\") pod \"4c23e4bb-1abd-4ff7-a1af-e71e32c579a1\" (UID: \"4c23e4bb-1abd-4ff7-a1af-e71e32c579a1\") " Sep 11 19:49:51.795274 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:51.795251 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c23e4bb-1abd-4ff7-a1af-e71e32c579a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4c23e4bb-1abd-4ff7-a1af-e71e32c579a1" (UID: "4c23e4bb-1abd-4ff7-a1af-e71e32c579a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:49:51.896455 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:51.896414 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c23e4bb-1abd-4ff7-a1af-e71e32c579a1-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:49:52.522957 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:52.522929 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" Sep 11 19:49:52.523139 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:52.522977 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7" event={"ID":"4c23e4bb-1abd-4ff7-a1af-e71e32c579a1","Type":"ContainerDied","Data":"71929a04f8115362e2901f3a0aed1a416330a39a26bba5ea9b520b22ff347149"} Sep 11 19:49:52.523139 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:52.523035 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71929a04f8115362e2901f3a0aed1a416330a39a26bba5ea9b520b22ff347149" Sep 11 19:49:58.866249 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:58.866160 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:49:59.435860 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.435822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr"] Sep 11 19:49:59.436061 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.436035 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" podUID="1c88f9e9-cb87-486f-a4eb-9dd5b07e2184" containerName="payload-processing" containerID="cri-o://2f413210e12da4c9218cf0e996d08ff3759885877ffb4b6e259edf4785e37a87" gracePeriod=30 Sep 11 19:49:59.444401 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.444365 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v"] Sep 11 19:49:59.444669 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.444636 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" podUID="9e32aba7-5a18-4429-b0c9-09bab80aa4f0" containerName="payload-pre-processing" containerID="cri-o://99d2e60eaf2dd28ffae2c7d452aa68a9a89885812d33ff56615eca75bd3c5fdc" gracePeriod=30 Sep 11 19:49:59.568280 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.568252 2559 generic.go:356] "Generic (PLEG): container finished" podID="9e32aba7-5a18-4429-b0c9-09bab80aa4f0" containerID="99d2e60eaf2dd28ffae2c7d452aa68a9a89885812d33ff56615eca75bd3c5fdc" exitCode=0 Sep 11 19:49:59.568419 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.568297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" event={"ID":"9e32aba7-5a18-4429-b0c9-09bab80aa4f0","Type":"ContainerDied","Data":"99d2e60eaf2dd28ffae2c7d452aa68a9a89885812d33ff56615eca75bd3c5fdc"} Sep 11 19:49:59.570492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.570468 2559 generic.go:356] "Generic (PLEG): container finished" podID="1c88f9e9-cb87-486f-a4eb-9dd5b07e2184" containerID="2f413210e12da4c9218cf0e996d08ff3759885877ffb4b6e259edf4785e37a87" exitCode=0 Sep 11 19:49:59.570624 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.570516 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" event={"ID":"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184","Type":"ContainerDied","Data":"2f413210e12da4c9218cf0e996d08ff3759885877ffb4b6e259edf4785e37a87"} Sep 11 19:49:59.606685 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.606640 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7"] Sep 11 19:49:59.626364 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.625671 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-02df91be-z6dk7"] Sep 11 19:49:59.698913 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.698845 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:49:59.733665 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.733640 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:49:59.782674 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.782641 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp4qx\" (UniqueName: \"kubernetes.io/projected/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-kube-api-access-gp4qx\") pod \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\" (UID: \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\") " Sep 11 19:49:59.782858 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.782716 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-plugins-config-volume\") pod \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\" (UID: \"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184\") " Sep 11 19:49:59.783079 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.783053 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1c88f9e9-cb87-486f-a4eb-9dd5b07e2184" (UID: "1c88f9e9-cb87-486f-a4eb-9dd5b07e2184"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:49:59.785057 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.785031 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-kube-api-access-gp4qx" (OuterVolumeSpecName: "kube-api-access-gp4qx") pod "1c88f9e9-cb87-486f-a4eb-9dd5b07e2184" (UID: "1c88f9e9-cb87-486f-a4eb-9dd5b07e2184"). InnerVolumeSpecName "kube-api-access-gp4qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:49:59.883606 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.883578 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-plugins-config-volume\") pod \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\" (UID: \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\") " Sep 11 19:49:59.883988 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.883629 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfgwd\" (UniqueName: \"kubernetes.io/projected/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-kube-api-access-hfgwd\") pod \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\" (UID: \"9e32aba7-5a18-4429-b0c9-09bab80aa4f0\") " Sep 11 19:49:59.883988 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.883807 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp4qx\" (UniqueName: \"kubernetes.io/projected/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-kube-api-access-gp4qx\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:49:59.883988 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.883819 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:49:59.883988 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.883924 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e32aba7-5a18-4429-b0c9-09bab80aa4f0" (UID: "9e32aba7-5a18-4429-b0c9-09bab80aa4f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:49:59.885916 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.885895 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-kube-api-access-hfgwd" (OuterVolumeSpecName: "kube-api-access-hfgwd") pod "9e32aba7-5a18-4429-b0c9-09bab80aa4f0" (UID: "9e32aba7-5a18-4429-b0c9-09bab80aa4f0"). InnerVolumeSpecName "kube-api-access-hfgwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:49:59.985299 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.985213 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:49:59.985299 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:49:59.985244 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfgwd\" (UniqueName: \"kubernetes.io/projected/9e32aba7-5a18-4429-b0c9-09bab80aa4f0-kube-api-access-hfgwd\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:50:00.578149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.578115 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" Sep 11 19:50:00.578149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.578142 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr" event={"ID":"1c88f9e9-cb87-486f-a4eb-9dd5b07e2184","Type":"ContainerDied","Data":"d31d53843b58c4f169b9e8f169529a0219d46efa4c0dd3fc7e62ba6207ea6355"} Sep 11 19:50:00.578412 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.578198 2559 scope.go:117] "RemoveContainer" containerID="2f413210e12da4c9218cf0e996d08ff3759885877ffb4b6e259edf4785e37a87" Sep 11 19:50:00.579738 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.579723 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" Sep 11 19:50:00.579826 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.579755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v" event={"ID":"9e32aba7-5a18-4429-b0c9-09bab80aa4f0","Type":"ContainerDied","Data":"f5dcad8a8cde6fffa7dea4b222697c6035a28a880098b10249054d5f364c1b75"} Sep 11 19:50:00.589661 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.589645 2559 scope.go:117] "RemoveContainer" containerID="99d2e60eaf2dd28ffae2c7d452aa68a9a89885812d33ff56615eca75bd3c5fdc" Sep 11 19:50:00.602032 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.602006 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr"] Sep 11 19:50:00.606528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.606493 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-02df91be-59644dccf7-8lptr"] Sep 11 19:50:00.617672 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.617647 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v"] Sep 11 19:50:00.622968 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:00.622947 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-02df91be-7c6656dbc8-xg96v"] Sep 11 19:50:01.224250 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:01.224216 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c88f9e9-cb87-486f-a4eb-9dd5b07e2184" path="/var/lib/kubelet/pods/1c88f9e9-cb87-486f-a4eb-9dd5b07e2184/volumes" Sep 11 19:50:01.224634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:01.224619 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c23e4bb-1abd-4ff7-a1af-e71e32c579a1" path="/var/lib/kubelet/pods/4c23e4bb-1abd-4ff7-a1af-e71e32c579a1/volumes" Sep 11 19:50:01.224984 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:01.224970 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e32aba7-5a18-4429-b0c9-09bab80aa4f0" path="/var/lib/kubelet/pods/9e32aba7-5a18-4429-b0c9-09bab80aa4f0/volumes" Sep 11 19:50:18.661634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:18.661598 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:50:43.152015 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.151980 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr"] Sep 11 19:50:43.152528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152414 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e32aba7-5a18-4429-b0c9-09bab80aa4f0" containerName="payload-pre-processing" Sep 11 19:50:43.152528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152426 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e32aba7-5a18-4429-b0c9-09bab80aa4f0" containerName="payload-pre-processing" Sep 11 19:50:43.152528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152453 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c23e4bb-1abd-4ff7-a1af-e71e32c579a1" containerName="revoke-keys" Sep 11 19:50:43.152528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152459 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c23e4bb-1abd-4ff7-a1af-e71e32c579a1" containerName="revoke-keys" Sep 11 19:50:43.152528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152473 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c88f9e9-cb87-486f-a4eb-9dd5b07e2184" containerName="payload-processing" Sep 11 19:50:43.152528 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152478 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c88f9e9-cb87-486f-a4eb-9dd5b07e2184" containerName="payload-processing" Sep 11 19:50:43.152727 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152548 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c88f9e9-cb87-486f-a4eb-9dd5b07e2184" containerName="payload-processing" Sep 11 19:50:43.152727 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152556 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e32aba7-5a18-4429-b0c9-09bab80aa4f0" containerName="payload-pre-processing" Sep 11 19:50:43.152727 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.152565 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c23e4bb-1abd-4ff7-a1af-e71e32c579a1" containerName="revoke-keys" Sep 11 19:50:43.155925 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.155905 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" Sep 11 19:50:43.159552 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.159484 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:50:43.159680 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.159500 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:50:43.162059 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.162029 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr"] Sep 11 19:50:43.296735 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.296703 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ae3db1-8153-410c-8503-21f080469cbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a29559-p2lxr\" (UID: \"08ae3db1-8153-410c-8503-21f080469cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" Sep 11 19:50:43.397762 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.397721 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ae3db1-8153-410c-8503-21f080469cbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a29559-p2lxr\" (UID: \"08ae3db1-8153-410c-8503-21f080469cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" Sep 11 19:50:43.398372 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.398346 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ae3db1-8153-410c-8503-21f080469cbd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-a29559-p2lxr\" (UID: \"08ae3db1-8153-410c-8503-21f080469cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" Sep 11 19:50:43.467279 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.467193 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" Sep 11 19:50:43.740714 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.740674 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr"] Sep 11 19:50:43.741906 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:50:43.741875 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ae3db1_8153_410c_8503_21f080469cbd.slice/crio-601b7dd0171e975ea73dab266607bc80bff753d4d98017254071060d5d5b2846 WatchSource:0}: Error finding container 601b7dd0171e975ea73dab266607bc80bff753d4d98017254071060d5d5b2846: Status 404 returned error can't find the container with id 601b7dd0171e975ea73dab266607bc80bff753d4d98017254071060d5d5b2846 Sep 11 19:50:43.876631 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:43.876594 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" event={"ID":"08ae3db1-8153-410c-8503-21f080469cbd","Type":"ContainerStarted","Data":"601b7dd0171e975ea73dab266607bc80bff753d4d98017254071060d5d5b2846"} Sep 11 19:50:44.888492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:44.888377 2559 generic.go:356] "Generic (PLEG): container finished" podID="08ae3db1-8153-410c-8503-21f080469cbd" containerID="4af4d4840daa258ab0d7e4b9f5ef16657a82b58d616e2d91b729992e712d06a1" exitCode=0 Sep 11 19:50:44.888492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:44.888467 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" event={"ID":"08ae3db1-8153-410c-8503-21f080469cbd","Type":"ContainerDied","Data":"4af4d4840daa258ab0d7e4b9f5ef16657a82b58d616e2d91b729992e712d06a1"} Sep 11 19:50:46.021307 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:46.021278 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" Sep 11 19:50:46.126516 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:46.126478 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ae3db1-8153-410c-8503-21f080469cbd-maas-api-service-ca\") pod \"08ae3db1-8153-410c-8503-21f080469cbd\" (UID: \"08ae3db1-8153-410c-8503-21f080469cbd\") " Sep 11 19:50:46.126866 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:46.126845 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ae3db1-8153-410c-8503-21f080469cbd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08ae3db1-8153-410c-8503-21f080469cbd" (UID: "08ae3db1-8153-410c-8503-21f080469cbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:50:46.228009 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:46.227912 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ae3db1-8153-410c-8503-21f080469cbd-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:50:46.903071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:46.903037 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" Sep 11 19:50:46.903266 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:46.903063 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr" event={"ID":"08ae3db1-8153-410c-8503-21f080469cbd","Type":"ContainerDied","Data":"601b7dd0171e975ea73dab266607bc80bff753d4d98017254071060d5d5b2846"} Sep 11 19:50:46.903266 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:46.903122 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="601b7dd0171e975ea73dab266607bc80bff753d4d98017254071060d5d5b2846" Sep 11 19:50:50.760725 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:50.760686 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:50:57.551554 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.551513 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb"] Sep 11 19:50:57.552149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.552116 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ae3db1-8153-410c-8503-21f080469cbd" containerName="revoke-keys" Sep 11 19:50:57.552149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.552150 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ae3db1-8153-410c-8503-21f080469cbd" containerName="revoke-keys" Sep 11 19:50:57.552298 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.552275 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ae3db1-8153-410c-8503-21f080469cbd" containerName="revoke-keys" Sep 11 19:50:57.555707 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.555685 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" Sep 11 19:50:57.558612 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.558593 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:50:57.560382 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.560364 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:50:57.574805 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.574780 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb"] Sep 11 19:50:57.635100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.635071 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831edb4e-1d7a-4db9-909a-09d755a776a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb\" (UID: \"831edb4e-1d7a-4db9-909a-09d755a776a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" Sep 11 19:50:57.736620 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.736577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831edb4e-1d7a-4db9-909a-09d755a776a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb\" (UID: \"831edb4e-1d7a-4db9-909a-09d755a776a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" Sep 11 19:50:57.737202 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.737182 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831edb4e-1d7a-4db9-909a-09d755a776a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb\" (UID: \"831edb4e-1d7a-4db9-909a-09d755a776a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" Sep 11 19:50:57.865690 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:57.865597 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" Sep 11 19:50:58.142070 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:58.142042 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb"] Sep 11 19:50:58.144582 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:50:58.144549 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831edb4e_1d7a_4db9_909a_09d755a776a7.slice/crio-16163a37db9cbaf850dea90d46e579abfdfaaadc368bbd5088cba2faa049a5b8 WatchSource:0}: Error finding container 16163a37db9cbaf850dea90d46e579abfdfaaadc368bbd5088cba2faa049a5b8: Status 404 returned error can't find the container with id 16163a37db9cbaf850dea90d46e579abfdfaaadc368bbd5088cba2faa049a5b8 Sep 11 19:50:58.988364 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:58.988330 2559 generic.go:356] "Generic (PLEG): container finished" podID="831edb4e-1d7a-4db9-909a-09d755a776a7" containerID="8ccadf30fbd7df08c9f873585cacb75d6099522a172b1386a58e2c87927902a2" exitCode=0 Sep 11 19:50:58.988752 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:58.988406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" event={"ID":"831edb4e-1d7a-4db9-909a-09d755a776a7","Type":"ContainerDied","Data":"8ccadf30fbd7df08c9f873585cacb75d6099522a172b1386a58e2c87927902a2"} Sep 11 19:50:58.988752 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:58.988480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" event={"ID":"831edb4e-1d7a-4db9-909a-09d755a776a7","Type":"ContainerStarted","Data":"16163a37db9cbaf850dea90d46e579abfdfaaadc368bbd5088cba2faa049a5b8"} Sep 11 19:50:59.283899 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.283862 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7"] Sep 11 19:50:59.284143 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.284116 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" podUID="72129686-8a3d-42cc-b711-4718199eaed7" containerName="payload-processing" containerID="cri-o://aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522" gracePeriod=30 Sep 11 19:50:59.290593 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.290563 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f"] Sep 11 19:50:59.290858 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.290790 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" podUID="48fdf415-2b4e-4b69-9ff6-87e234ba08cf" containerName="payload-pre-processing" containerID="cri-o://92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a" gracePeriod=30 Sep 11 19:50:59.473754 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.473717 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr"] Sep 11 19:50:59.478492 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.478465 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-a29559-p2lxr"] Sep 11 19:50:59.578275 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.578250 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:50:59.581083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.581064 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:50:59.655662 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.655629 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wg48\" (UniqueName: \"kubernetes.io/projected/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-kube-api-access-5wg48\") pod \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\" (UID: \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\") " Sep 11 19:50:59.655662 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.655666 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72129686-8a3d-42cc-b711-4718199eaed7-plugins-config-volume\") pod \"72129686-8a3d-42cc-b711-4718199eaed7\" (UID: \"72129686-8a3d-42cc-b711-4718199eaed7\") " Sep 11 19:50:59.655857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.655717 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p5hv\" (UniqueName: \"kubernetes.io/projected/72129686-8a3d-42cc-b711-4718199eaed7-kube-api-access-8p5hv\") pod \"72129686-8a3d-42cc-b711-4718199eaed7\" (UID: \"72129686-8a3d-42cc-b711-4718199eaed7\") " Sep 11 19:50:59.655857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.655795 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-plugins-config-volume\") pod \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\" (UID: \"48fdf415-2b4e-4b69-9ff6-87e234ba08cf\") " Sep 11 19:50:59.656195 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.656168 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72129686-8a3d-42cc-b711-4718199eaed7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "72129686-8a3d-42cc-b711-4718199eaed7" (UID: "72129686-8a3d-42cc-b711-4718199eaed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:50:59.656313 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.656201 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "48fdf415-2b4e-4b69-9ff6-87e234ba08cf" (UID: "48fdf415-2b4e-4b69-9ff6-87e234ba08cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:50:59.657921 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.657901 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72129686-8a3d-42cc-b711-4718199eaed7-kube-api-access-8p5hv" (OuterVolumeSpecName: "kube-api-access-8p5hv") pod "72129686-8a3d-42cc-b711-4718199eaed7" (UID: "72129686-8a3d-42cc-b711-4718199eaed7"). InnerVolumeSpecName "kube-api-access-8p5hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:50:59.658034 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.658016 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-kube-api-access-5wg48" (OuterVolumeSpecName: "kube-api-access-5wg48") pod "48fdf415-2b4e-4b69-9ff6-87e234ba08cf" (UID: "48fdf415-2b4e-4b69-9ff6-87e234ba08cf"). InnerVolumeSpecName "kube-api-access-5wg48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:50:59.756909 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.756873 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:50:59.756909 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.756904 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wg48\" (UniqueName: \"kubernetes.io/projected/48fdf415-2b4e-4b69-9ff6-87e234ba08cf-kube-api-access-5wg48\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:50:59.756909 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.756914 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72129686-8a3d-42cc-b711-4718199eaed7-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:50:59.757150 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.756922 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5hv\" (UniqueName: \"kubernetes.io/projected/72129686-8a3d-42cc-b711-4718199eaed7-kube-api-access-8p5hv\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:50:59.995920 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.995888 2559 generic.go:356] "Generic (PLEG): container finished" podID="72129686-8a3d-42cc-b711-4718199eaed7" containerID="aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522" exitCode=0 Sep 11 19:50:59.996374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.995938 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" Sep 11 19:50:59.996374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.995979 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" event={"ID":"72129686-8a3d-42cc-b711-4718199eaed7","Type":"ContainerDied","Data":"aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522"} Sep 11 19:50:59.996374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.996039 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7" event={"ID":"72129686-8a3d-42cc-b711-4718199eaed7","Type":"ContainerDied","Data":"1e0ea7559b4a44ff6acfe2bbfaaf6da567882b53d9210a8f38af1922ff6a87be"} Sep 11 19:50:59.996374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.996067 2559 scope.go:117] "RemoveContainer" containerID="aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522" Sep 11 19:50:59.997877 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.997851 2559 generic.go:356] "Generic (PLEG): container finished" podID="48fdf415-2b4e-4b69-9ff6-87e234ba08cf" containerID="92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a" exitCode=0 Sep 11 19:50:59.998000 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.997909 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" Sep 11 19:50:59.998000 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.997971 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" event={"ID":"48fdf415-2b4e-4b69-9ff6-87e234ba08cf","Type":"ContainerDied","Data":"92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a"} Sep 11 19:50:59.998125 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:50:59.998012 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f" event={"ID":"48fdf415-2b4e-4b69-9ff6-87e234ba08cf","Type":"ContainerDied","Data":"f45b0e793c95b52695c89ec6cf9de43d6baf2dec1d97bf4f3bc0cb7d45d20c54"} Sep 11 19:51:00.008927 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.008901 2559 scope.go:117] "RemoveContainer" containerID="aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522" Sep 11 19:51:00.009260 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:00.009235 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522\": container with ID starting with aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522 not found: ID does not exist" containerID="aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522" Sep 11 19:51:00.009358 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.009271 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522"} err="failed to get container status \"aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522\": rpc error: code = NotFound desc = could not find container \"aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522\": container with ID starting with aaefb4dd231934e5eaf69b9dbb8552a60894395609767750b21b31164a5e1522 not found: ID does not exist" Sep 11 19:51:00.009358 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.009292 2559 scope.go:117] "RemoveContainer" containerID="92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a" Sep 11 19:51:00.024293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.024193 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7"] Sep 11 19:51:00.024293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.024269 2559 scope.go:117] "RemoveContainer" containerID="92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a" Sep 11 19:51:00.024947 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:00.024923 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a\": container with ID starting with 92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a not found: ID does not exist" containerID="92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a" Sep 11 19:51:00.025039 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.024955 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a"} err="failed to get container status \"92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a\": rpc error: code = NotFound desc = could not find container \"92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a\": container with ID starting with 92a279b16615aa2d189c52abb4e00fc86fe356ee0cd50e3f37106a5ba6b9fc4a not found: ID does not exist" Sep 11 19:51:00.027325 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.027305 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-a29559-55c4c594f7-mpdm7"] Sep 11 19:51:00.038138 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.038080 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f"] Sep 11 19:51:00.041356 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.041335 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-a29559-8d44cc5f8-pl58f"] Sep 11 19:51:00.128886 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.128860 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" Sep 11 19:51:00.161207 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.161176 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831edb4e-1d7a-4db9-909a-09d755a776a7-maas-api-service-ca\") pod \"831edb4e-1d7a-4db9-909a-09d755a776a7\" (UID: \"831edb4e-1d7a-4db9-909a-09d755a776a7\") " Sep 11 19:51:00.161548 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.161526 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831edb4e-1d7a-4db9-909a-09d755a776a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "831edb4e-1d7a-4db9-909a-09d755a776a7" (UID: "831edb4e-1d7a-4db9-909a-09d755a776a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:51:00.240452 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.240391 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb"] Sep 11 19:51:00.240892 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.240874 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48fdf415-2b4e-4b69-9ff6-87e234ba08cf" containerName="payload-pre-processing" Sep 11 19:51:00.240982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.240896 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="48fdf415-2b4e-4b69-9ff6-87e234ba08cf" containerName="payload-pre-processing" Sep 11 19:51:00.240982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.240928 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72129686-8a3d-42cc-b711-4718199eaed7" containerName="payload-processing" Sep 11 19:51:00.240982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.240937 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="72129686-8a3d-42cc-b711-4718199eaed7" containerName="payload-processing" Sep 11 19:51:00.240982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.240946 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="831edb4e-1d7a-4db9-909a-09d755a776a7" containerName="revoke-keys" Sep 11 19:51:00.240982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.240954 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="831edb4e-1d7a-4db9-909a-09d755a776a7" containerName="revoke-keys" Sep 11 19:51:00.241222 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.241086 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="72129686-8a3d-42cc-b711-4718199eaed7" containerName="payload-processing" Sep 11 19:51:00.241222 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.241100 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="831edb4e-1d7a-4db9-909a-09d755a776a7" containerName="revoke-keys" Sep 11 19:51:00.241222 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.241111 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="48fdf415-2b4e-4b69-9ff6-87e234ba08cf" containerName="payload-pre-processing" Sep 11 19:51:00.244365 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.244344 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:00.246972 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.246910 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d14d7e\"" Sep 11 19:51:00.247067 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.247050 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d14d7e-dockercfg-gps5q\"" Sep 11 19:51:00.253400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.253376 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb"] Sep 11 19:51:00.259814 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.259788 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc"] Sep 11 19:51:00.262473 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.262423 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831edb4e-1d7a-4db9-909a-09d755a776a7-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:00.263189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.263168 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:00.289712 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.289685 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc"] Sep 11 19:51:00.363262 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.363205 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78kkw\" (UniqueName: \"kubernetes.io/projected/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-kube-api-access-78kkw\") pod \"payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb\" (UID: \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:00.363474 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.363281 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztxtq\" (UniqueName: \"kubernetes.io/projected/371f23d0-9226-4fcd-b670-2f54b406e3b5-kube-api-access-ztxtq\") pod \"payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc\" (UID: \"371f23d0-9226-4fcd-b670-2f54b406e3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:00.363474 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.363357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb\" (UID: \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:00.363474 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.363394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371f23d0-9226-4fcd-b670-2f54b406e3b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc\" (UID: \"371f23d0-9226-4fcd-b670-2f54b406e3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:00.464060 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.464019 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kkw\" (UniqueName: \"kubernetes.io/projected/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-kube-api-access-78kkw\") pod \"payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb\" (UID: \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:00.464236 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.464074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztxtq\" (UniqueName: \"kubernetes.io/projected/371f23d0-9226-4fcd-b670-2f54b406e3b5-kube-api-access-ztxtq\") pod \"payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc\" (UID: \"371f23d0-9226-4fcd-b670-2f54b406e3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:00.464236 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.464126 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb\" (UID: \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:00.464236 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.464147 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371f23d0-9226-4fcd-b670-2f54b406e3b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc\" (UID: \"371f23d0-9226-4fcd-b670-2f54b406e3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:00.464866 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.464844 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb\" (UID: \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:00.464908 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.464845 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371f23d0-9226-4fcd-b670-2f54b406e3b5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc\" (UID: \"371f23d0-9226-4fcd-b670-2f54b406e3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:00.473071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.473038 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kkw\" (UniqueName: \"kubernetes.io/projected/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-kube-api-access-78kkw\") pod \"payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb\" (UID: \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:00.473989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.473970 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztxtq\" (UniqueName: \"kubernetes.io/projected/371f23d0-9226-4fcd-b670-2f54b406e3b5-kube-api-access-ztxtq\") pod \"payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc\" (UID: \"371f23d0-9226-4fcd-b670-2f54b406e3b5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:00.556049 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.555952 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:00.574392 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.574362 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:00.843832 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.843800 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb"] Sep 11 19:51:00.845351 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:00.845320 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa1e6743_3084_43c1_b4c9_ba2f6ae09012.slice/crio-6cf83739e8e7c306fc9f9495d2a9d86e1537f926d428fde0bc7f1df0e7c88a5c WatchSource:0}: Error finding container 6cf83739e8e7c306fc9f9495d2a9d86e1537f926d428fde0bc7f1df0e7c88a5c: Status 404 returned error can't find the container with id 6cf83739e8e7c306fc9f9495d2a9d86e1537f926d428fde0bc7f1df0e7c88a5c Sep 11 19:51:00.864010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:00.863985 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc"] Sep 11 19:51:00.865432 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:00.865409 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371f23d0_9226_4fcd_b670_2f54b406e3b5.slice/crio-3c60c9bbcc0d7e246e5ee057c9d148921e90c9999f608cbf48483995f0177612 WatchSource:0}: Error finding container 3c60c9bbcc0d7e246e5ee057c9d148921e90c9999f608cbf48483995f0177612: Status 404 returned error can't find the container with id 3c60c9bbcc0d7e246e5ee057c9d148921e90c9999f608cbf48483995f0177612 Sep 11 19:51:01.009401 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.009363 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" event={"ID":"371f23d0-9226-4fcd-b670-2f54b406e3b5","Type":"ContainerStarted","Data":"366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d"} Sep 11 19:51:01.009957 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.009426 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" event={"ID":"371f23d0-9226-4fcd-b670-2f54b406e3b5","Type":"ContainerStarted","Data":"3c60c9bbcc0d7e246e5ee057c9d148921e90c9999f608cbf48483995f0177612"} Sep 11 19:51:01.010346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.010107 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:01.011956 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.011919 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" Sep 11 19:51:01.011956 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.011946 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb" event={"ID":"831edb4e-1d7a-4db9-909a-09d755a776a7","Type":"ContainerDied","Data":"16163a37db9cbaf850dea90d46e579abfdfaaadc368bbd5088cba2faa049a5b8"} Sep 11 19:51:01.012126 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.011993 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16163a37db9cbaf850dea90d46e579abfdfaaadc368bbd5088cba2faa049a5b8" Sep 11 19:51:01.015512 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.015480 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" event={"ID":"fa1e6743-3084-43c1-b4c9-ba2f6ae09012","Type":"ContainerStarted","Data":"4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240"} Sep 11 19:51:01.015618 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.015524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" event={"ID":"fa1e6743-3084-43c1-b4c9-ba2f6ae09012","Type":"ContainerStarted","Data":"6cf83739e8e7c306fc9f9495d2a9d86e1537f926d428fde0bc7f1df0e7c88a5c"} Sep 11 19:51:01.015692 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.015645 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:01.026774 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.026736 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" podStartSLOduration=1.026723107 podStartE2EDuration="1.026723107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:01.025681406 +0000 UTC m=+1494.298188607" watchObservedRunningTime="2026-09-11 19:51:01.026723107 +0000 UTC m=+1494.299230307" Sep 11 19:51:01.044597 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.044554 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" podStartSLOduration=1.044541896 podStartE2EDuration="1.044541896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:01.04390757 +0000 UTC m=+1494.316414770" watchObservedRunningTime="2026-09-11 19:51:01.044541896 +0000 UTC m=+1494.317049096" Sep 11 19:51:01.224267 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.224186 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ae3db1-8153-410c-8503-21f080469cbd" path="/var/lib/kubelet/pods/08ae3db1-8153-410c-8503-21f080469cbd/volumes" Sep 11 19:51:01.224580 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.224565 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48fdf415-2b4e-4b69-9ff6-87e234ba08cf" path="/var/lib/kubelet/pods/48fdf415-2b4e-4b69-9ff6-87e234ba08cf/volumes" Sep 11 19:51:01.224875 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:01.224863 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72129686-8a3d-42cc-b711-4718199eaed7" path="/var/lib/kubelet/pods/72129686-8a3d-42cc-b711-4718199eaed7/volumes" Sep 11 19:51:08.345239 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.345193 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7"] Sep 11 19:51:08.350263 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.350234 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-ec80ad-29819265-zz5j7"] Sep 11 19:51:08.370282 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.370238 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr"] Sep 11 19:51:08.370555 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.370530 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" podUID="f2a31aba-d6ac-451a-9fb0-b34e81f518c2" containerName="payload-processing" containerID="cri-o://3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6" gracePeriod=30 Sep 11 19:51:08.380274 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.380244 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87"] Sep 11 19:51:08.380686 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.380651 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" podUID="0bca2e9a-8235-4622-9ba5-c771b427917a" containerName="payload-pre-processing" containerID="cri-o://8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83" gracePeriod=30 Sep 11 19:51:08.613729 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.613699 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb"] Sep 11 19:51:08.622110 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.622083 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ec80ad-jdbpb"] Sep 11 19:51:08.637653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.637623 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:51:08.666560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.666533 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:51:08.749362 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.749311 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bca2e9a-8235-4622-9ba5-c771b427917a-plugins-config-volume\") pod \"0bca2e9a-8235-4622-9ba5-c771b427917a\" (UID: \"0bca2e9a-8235-4622-9ba5-c771b427917a\") " Sep 11 19:51:08.749362 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.749373 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-plugins-config-volume\") pod \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\" (UID: \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\") " Sep 11 19:51:08.749607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.749397 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztpbv\" (UniqueName: \"kubernetes.io/projected/0bca2e9a-8235-4622-9ba5-c771b427917a-kube-api-access-ztpbv\") pod \"0bca2e9a-8235-4622-9ba5-c771b427917a\" (UID: \"0bca2e9a-8235-4622-9ba5-c771b427917a\") " Sep 11 19:51:08.749607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.749432 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7zjt\" (UniqueName: \"kubernetes.io/projected/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-kube-api-access-s7zjt\") pod \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\" (UID: \"f2a31aba-d6ac-451a-9fb0-b34e81f518c2\") " Sep 11 19:51:08.750176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.749835 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2a31aba-d6ac-451a-9fb0-b34e81f518c2" (UID: "f2a31aba-d6ac-451a-9fb0-b34e81f518c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:51:08.750176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.749835 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bca2e9a-8235-4622-9ba5-c771b427917a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0bca2e9a-8235-4622-9ba5-c771b427917a" (UID: "0bca2e9a-8235-4622-9ba5-c771b427917a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:51:08.751846 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.751817 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-kube-api-access-s7zjt" (OuterVolumeSpecName: "kube-api-access-s7zjt") pod "f2a31aba-d6ac-451a-9fb0-b34e81f518c2" (UID: "f2a31aba-d6ac-451a-9fb0-b34e81f518c2"). InnerVolumeSpecName "kube-api-access-s7zjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:51:08.751846 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.751824 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bca2e9a-8235-4622-9ba5-c771b427917a-kube-api-access-ztpbv" (OuterVolumeSpecName: "kube-api-access-ztpbv") pod "0bca2e9a-8235-4622-9ba5-c771b427917a" (UID: "0bca2e9a-8235-4622-9ba5-c771b427917a"). InnerVolumeSpecName "kube-api-access-ztpbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:51:08.851195 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.851159 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:08.851195 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.851189 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztpbv\" (UniqueName: \"kubernetes.io/projected/0bca2e9a-8235-4622-9ba5-c771b427917a-kube-api-access-ztpbv\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:08.851195 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.851198 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7zjt\" (UniqueName: \"kubernetes.io/projected/f2a31aba-d6ac-451a-9fb0-b34e81f518c2-kube-api-access-s7zjt\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:08.851421 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:08.851206 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bca2e9a-8235-4622-9ba5-c771b427917a-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:09.069344 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.069304 2559 generic.go:356] "Generic (PLEG): container finished" podID="f2a31aba-d6ac-451a-9fb0-b34e81f518c2" containerID="3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6" exitCode=0 Sep 11 19:51:09.069550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.069351 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" Sep 11 19:51:09.069550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.069389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" event={"ID":"f2a31aba-d6ac-451a-9fb0-b34e81f518c2","Type":"ContainerDied","Data":"3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6"} Sep 11 19:51:09.069550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.069468 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr" event={"ID":"f2a31aba-d6ac-451a-9fb0-b34e81f518c2","Type":"ContainerDied","Data":"e2b817c9eacd4a2d70ecfd267af134458e9c7f998040a969f83a378c986537be"} Sep 11 19:51:09.069550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.069491 2559 scope.go:117] "RemoveContainer" containerID="3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6" Sep 11 19:51:09.071028 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.071005 2559 generic.go:356] "Generic (PLEG): container finished" podID="0bca2e9a-8235-4622-9ba5-c771b427917a" containerID="8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83" exitCode=0 Sep 11 19:51:09.071145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.071053 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" Sep 11 19:51:09.071145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.071075 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" event={"ID":"0bca2e9a-8235-4622-9ba5-c771b427917a","Type":"ContainerDied","Data":"8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83"} Sep 11 19:51:09.071145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.071114 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87" event={"ID":"0bca2e9a-8235-4622-9ba5-c771b427917a","Type":"ContainerDied","Data":"061d9006f763e8b0d8cb9138894a6ef9d81c876b07954df165a74e8f93ce23ba"} Sep 11 19:51:09.079468 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.079431 2559 scope.go:117] "RemoveContainer" containerID="3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6" Sep 11 19:51:09.079795 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:09.079770 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6\": container with ID starting with 3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6 not found: ID does not exist" containerID="3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6" Sep 11 19:51:09.079860 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.079809 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6"} err="failed to get container status \"3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6\": rpc error: code = NotFound desc = could not find container \"3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6\": container with ID starting with 3b103e478533c6bd884f0b825cf635b2f3adc98238aa4c17d35be9650e0b68d6 not found: ID does not exist" Sep 11 19:51:09.079860 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.079831 2559 scope.go:117] "RemoveContainer" containerID="8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83" Sep 11 19:51:09.088259 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.088245 2559 scope.go:117] "RemoveContainer" containerID="8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83" Sep 11 19:51:09.088560 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:09.088543 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83\": container with ID starting with 8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83 not found: ID does not exist" containerID="8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83" Sep 11 19:51:09.088621 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.088571 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83"} err="failed to get container status \"8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83\": rpc error: code = NotFound desc = could not find container \"8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83\": container with ID starting with 8940395c83263e202052c1d2f3e9fe23706972ac85d22ed58c7f9b92bed4bd83 not found: ID does not exist" Sep 11 19:51:09.097253 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.097228 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87"] Sep 11 19:51:09.102763 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.102739 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ec80ad-699559ffc4-nrx87"] Sep 11 19:51:09.113758 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.113728 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr"] Sep 11 19:51:09.116120 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.116101 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ec80ad-766949768c-7gshr"] Sep 11 19:51:09.224626 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.224584 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bca2e9a-8235-4622-9ba5-c771b427917a" path="/var/lib/kubelet/pods/0bca2e9a-8235-4622-9ba5-c771b427917a/volumes" Sep 11 19:51:09.225022 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.225003 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="831edb4e-1d7a-4db9-909a-09d755a776a7" path="/var/lib/kubelet/pods/831edb4e-1d7a-4db9-909a-09d755a776a7/volumes" Sep 11 19:51:09.225313 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.225295 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db702102-1ea1-4244-9b8e-149f1a94f2f6" path="/var/lib/kubelet/pods/db702102-1ea1-4244-9b8e-149f1a94f2f6/volumes" Sep 11 19:51:09.225675 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:09.225656 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a31aba-d6ac-451a-9fb0-b34e81f518c2" path="/var/lib/kubelet/pods/f2a31aba-d6ac-451a-9fb0-b34e81f518c2/volumes" Sep 11 19:51:12.024652 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:12.024618 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:51:13.031481 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:13.031425 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:51:15.275024 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:15.274989 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:51:16.514456 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.514397 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm"] Sep 11 19:51:16.514833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.514817 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a31aba-d6ac-451a-9fb0-b34e81f518c2" containerName="payload-processing" Sep 11 19:51:16.514833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.514828 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a31aba-d6ac-451a-9fb0-b34e81f518c2" containerName="payload-processing" Sep 11 19:51:16.514904 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.514853 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bca2e9a-8235-4622-9ba5-c771b427917a" containerName="payload-pre-processing" Sep 11 19:51:16.514904 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.514859 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bca2e9a-8235-4622-9ba5-c771b427917a" containerName="payload-pre-processing" Sep 11 19:51:16.514970 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.514921 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bca2e9a-8235-4622-9ba5-c771b427917a" containerName="payload-pre-processing" Sep 11 19:51:16.514970 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.514932 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a31aba-d6ac-451a-9fb0-b34e81f518c2" containerName="payload-processing" Sep 11 19:51:16.519524 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.519500 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" Sep 11 19:51:16.522257 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.522237 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:51:16.522394 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.522339 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:51:16.528275 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.527739 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm"] Sep 11 19:51:16.628176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.628141 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25955265-4498-4038-885d-d5b6badbd241-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-872425-564sm\" (UID: \"25955265-4498-4038-885d-d5b6badbd241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" Sep 11 19:51:16.728890 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.728843 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25955265-4498-4038-885d-d5b6badbd241-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-872425-564sm\" (UID: \"25955265-4498-4038-885d-d5b6badbd241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" Sep 11 19:51:16.729637 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.729613 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25955265-4498-4038-885d-d5b6badbd241-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-872425-564sm\" (UID: \"25955265-4498-4038-885d-d5b6badbd241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" Sep 11 19:51:16.832889 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:16.832806 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" Sep 11 19:51:17.105800 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:17.105769 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm"] Sep 11 19:51:17.106262 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:17.106229 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25955265_4498_4038_885d_d5b6badbd241.slice/crio-254759b5487d2675d07076cf4f0ef10f728ddc8a1ed5b0cd7a633cdd14e81fca WatchSource:0}: Error finding container 254759b5487d2675d07076cf4f0ef10f728ddc8a1ed5b0cd7a633cdd14e81fca: Status 404 returned error can't find the container with id 254759b5487d2675d07076cf4f0ef10f728ddc8a1ed5b0cd7a633cdd14e81fca Sep 11 19:51:17.136936 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:17.136911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" event={"ID":"25955265-4498-4038-885d-d5b6badbd241","Type":"ContainerStarted","Data":"254759b5487d2675d07076cf4f0ef10f728ddc8a1ed5b0cd7a633cdd14e81fca"} Sep 11 19:51:18.144967 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:18.144930 2559 generic.go:356] "Generic (PLEG): container finished" podID="25955265-4498-4038-885d-d5b6badbd241" containerID="c3ae5b33599e222ea7b0d16dfc745473db56acfb5741caaf2296daa9435fa5f3" exitCode=0 Sep 11 19:51:18.145386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:18.145015 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" event={"ID":"25955265-4498-4038-885d-d5b6badbd241","Type":"ContainerDied","Data":"c3ae5b33599e222ea7b0d16dfc745473db56acfb5741caaf2296daa9435fa5f3"} Sep 11 19:51:19.296641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:19.296612 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" Sep 11 19:51:19.453484 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:19.453369 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25955265-4498-4038-885d-d5b6badbd241-maas-api-service-ca\") pod \"25955265-4498-4038-885d-d5b6badbd241\" (UID: \"25955265-4498-4038-885d-d5b6badbd241\") " Sep 11 19:51:19.453762 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:19.453737 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25955265-4498-4038-885d-d5b6badbd241-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25955265-4498-4038-885d-d5b6badbd241" (UID: "25955265-4498-4038-885d-d5b6badbd241"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:51:19.554320 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:19.554278 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25955265-4498-4038-885d-d5b6badbd241-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:20.160471 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:20.160423 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" Sep 11 19:51:20.160656 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:20.160488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm" event={"ID":"25955265-4498-4038-885d-d5b6badbd241","Type":"ContainerDied","Data":"254759b5487d2675d07076cf4f0ef10f728ddc8a1ed5b0cd7a633cdd14e81fca"} Sep 11 19:51:20.160656 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:20.160536 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="254759b5487d2675d07076cf4f0ef10f728ddc8a1ed5b0cd7a633cdd14e81fca" Sep 11 19:51:22.036666 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.036631 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l"] Sep 11 19:51:22.037070 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.037057 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25955265-4498-4038-885d-d5b6badbd241" containerName="revoke-keys" Sep 11 19:51:22.037123 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.037072 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="25955265-4498-4038-885d-d5b6badbd241" containerName="revoke-keys" Sep 11 19:51:22.037165 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.037157 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="25955265-4498-4038-885d-d5b6badbd241" containerName="revoke-keys" Sep 11 19:51:22.040322 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.040306 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:22.043217 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.043186 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-58f57d-dockercfg-5jqxw\"" Sep 11 19:51:22.043327 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.043221 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-58f57d\"" Sep 11 19:51:22.052542 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.052519 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l"] Sep 11 19:51:22.070317 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.070290 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w"] Sep 11 19:51:22.074109 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.074092 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:22.084952 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.084870 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w"] Sep 11 19:51:22.178865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.178832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d66807-fd8e-42e0-801c-cb27842c9007-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l\" (UID: \"17d66807-fd8e-42e0-801c-cb27842c9007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:22.179043 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.178892 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2jld\" (UniqueName: \"kubernetes.io/projected/17d66807-fd8e-42e0-801c-cb27842c9007-kube-api-access-m2jld\") pod \"payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l\" (UID: \"17d66807-fd8e-42e0-801c-cb27842c9007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:22.179043 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.178966 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27cxj\" (UniqueName: \"kubernetes.io/projected/e6c566ba-c83a-4133-93f3-838605ce413b-kube-api-access-27cxj\") pod \"payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w\" (UID: \"e6c566ba-c83a-4133-93f3-838605ce413b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:22.179043 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.179026 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c566ba-c83a-4133-93f3-838605ce413b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w\" (UID: \"e6c566ba-c83a-4133-93f3-838605ce413b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:22.279983 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.279942 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2jld\" (UniqueName: \"kubernetes.io/projected/17d66807-fd8e-42e0-801c-cb27842c9007-kube-api-access-m2jld\") pod \"payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l\" (UID: \"17d66807-fd8e-42e0-801c-cb27842c9007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:22.280188 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.279991 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27cxj\" (UniqueName: \"kubernetes.io/projected/e6c566ba-c83a-4133-93f3-838605ce413b-kube-api-access-27cxj\") pod \"payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w\" (UID: \"e6c566ba-c83a-4133-93f3-838605ce413b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:22.280188 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.280039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c566ba-c83a-4133-93f3-838605ce413b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w\" (UID: \"e6c566ba-c83a-4133-93f3-838605ce413b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:22.280310 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.280185 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d66807-fd8e-42e0-801c-cb27842c9007-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l\" (UID: \"17d66807-fd8e-42e0-801c-cb27842c9007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:22.280756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.280738 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c566ba-c83a-4133-93f3-838605ce413b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w\" (UID: \"e6c566ba-c83a-4133-93f3-838605ce413b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:22.280874 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.280855 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d66807-fd8e-42e0-801c-cb27842c9007-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l\" (UID: \"17d66807-fd8e-42e0-801c-cb27842c9007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:22.288965 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.288901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27cxj\" (UniqueName: \"kubernetes.io/projected/e6c566ba-c83a-4133-93f3-838605ce413b-kube-api-access-27cxj\") pod \"payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w\" (UID: \"e6c566ba-c83a-4133-93f3-838605ce413b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:22.288965 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.288901 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2jld\" (UniqueName: \"kubernetes.io/projected/17d66807-fd8e-42e0-801c-cb27842c9007-kube-api-access-m2jld\") pod \"payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l\" (UID: \"17d66807-fd8e-42e0-801c-cb27842c9007\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:22.351718 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.351683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:22.385317 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.385284 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:22.632127 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.632104 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l"] Sep 11 19:51:22.634258 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:22.634235 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d66807_fd8e_42e0_801c_cb27842c9007.slice/crio-648827aa28c116344e360c5e04042aa92bbecd9d0c33c65dda517bcdb18bc326 WatchSource:0}: Error finding container 648827aa28c116344e360c5e04042aa92bbecd9d0c33c65dda517bcdb18bc326: Status 404 returned error can't find the container with id 648827aa28c116344e360c5e04042aa92bbecd9d0c33c65dda517bcdb18bc326 Sep 11 19:51:22.670353 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:22.670327 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w"] Sep 11 19:51:22.671592 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:22.671564 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c566ba_c83a_4133_93f3_838605ce413b.slice/crio-e7e13c9654207ff9593e1b59b8d6d374f86f82e2c20ab721597b43dc3bba872d WatchSource:0}: Error finding container e7e13c9654207ff9593e1b59b8d6d374f86f82e2c20ab721597b43dc3bba872d: Status 404 returned error can't find the container with id e7e13c9654207ff9593e1b59b8d6d374f86f82e2c20ab721597b43dc3bba872d Sep 11 19:51:23.186977 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:23.186946 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" event={"ID":"e6c566ba-c83a-4133-93f3-838605ce413b","Type":"ContainerStarted","Data":"67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1"} Sep 11 19:51:23.187497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:23.187000 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" event={"ID":"e6c566ba-c83a-4133-93f3-838605ce413b","Type":"ContainerStarted","Data":"e7e13c9654207ff9593e1b59b8d6d374f86f82e2c20ab721597b43dc3bba872d"} Sep 11 19:51:23.187497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:23.187153 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:23.189117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:23.189088 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" event={"ID":"17d66807-fd8e-42e0-801c-cb27842c9007","Type":"ContainerStarted","Data":"1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33"} Sep 11 19:51:23.189259 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:23.189145 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" event={"ID":"17d66807-fd8e-42e0-801c-cb27842c9007","Type":"ContainerStarted","Data":"648827aa28c116344e360c5e04042aa92bbecd9d0c33c65dda517bcdb18bc326"} Sep 11 19:51:23.189259 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:23.189174 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:23.217043 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:23.216992 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" podStartSLOduration=1.216976026 podStartE2EDuration="1.216976026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:23.2148911 +0000 UTC m=+1516.487398295" watchObservedRunningTime="2026-09-11 19:51:23.216976026 +0000 UTC m=+1516.489483226" Sep 11 19:51:23.233394 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:23.233346 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" podStartSLOduration=1.233330717 podStartE2EDuration="1.233330717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:23.233123131 +0000 UTC m=+1516.505630334" watchObservedRunningTime="2026-09-11 19:51:23.233330717 +0000 UTC m=+1516.505837963" Sep 11 19:51:27.263972 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.263936 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27"] Sep 11 19:51:27.264371 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.264195 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" podUID="4435c463-fcd7-494d-9d19-34270da006f1" containerName="payload-processing" containerID="cri-o://423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9" gracePeriod=30 Sep 11 19:51:27.271087 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.271052 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw"] Sep 11 19:51:27.271304 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.271279 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" podUID="32cb86bd-84ad-4a2f-a884-6f43c41b75de" containerName="payload-pre-processing" containerID="cri-o://c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a" gracePeriod=30 Sep 11 19:51:27.522458 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.522370 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:51:27.528911 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.528888 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm"] Sep 11 19:51:27.533944 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.533917 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-872425-564sm"] Sep 11 19:51:27.557110 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.557089 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:51:27.641780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.641748 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4435c463-fcd7-494d-9d19-34270da006f1-plugins-config-volume\") pod \"4435c463-fcd7-494d-9d19-34270da006f1\" (UID: \"4435c463-fcd7-494d-9d19-34270da006f1\") " Sep 11 19:51:27.641951 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.641861 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfpmf\" (UniqueName: \"kubernetes.io/projected/4435c463-fcd7-494d-9d19-34270da006f1-kube-api-access-jfpmf\") pod \"4435c463-fcd7-494d-9d19-34270da006f1\" (UID: \"4435c463-fcd7-494d-9d19-34270da006f1\") " Sep 11 19:51:27.642130 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.642096 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4435c463-fcd7-494d-9d19-34270da006f1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4435c463-fcd7-494d-9d19-34270da006f1" (UID: "4435c463-fcd7-494d-9d19-34270da006f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:51:27.644184 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.644161 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4435c463-fcd7-494d-9d19-34270da006f1-kube-api-access-jfpmf" (OuterVolumeSpecName: "kube-api-access-jfpmf") pod "4435c463-fcd7-494d-9d19-34270da006f1" (UID: "4435c463-fcd7-494d-9d19-34270da006f1"). InnerVolumeSpecName "kube-api-access-jfpmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:51:27.742952 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.742920 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gkcr\" (UniqueName: \"kubernetes.io/projected/32cb86bd-84ad-4a2f-a884-6f43c41b75de-kube-api-access-2gkcr\") pod \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\" (UID: \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\") " Sep 11 19:51:27.743156 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.742996 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cb86bd-84ad-4a2f-a884-6f43c41b75de-plugins-config-volume\") pod \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\" (UID: \"32cb86bd-84ad-4a2f-a884-6f43c41b75de\") " Sep 11 19:51:27.743284 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.743266 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfpmf\" (UniqueName: \"kubernetes.io/projected/4435c463-fcd7-494d-9d19-34270da006f1-kube-api-access-jfpmf\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:27.743345 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.743291 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4435c463-fcd7-494d-9d19-34270da006f1-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:27.743401 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.743378 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32cb86bd-84ad-4a2f-a884-6f43c41b75de-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "32cb86bd-84ad-4a2f-a884-6f43c41b75de" (UID: "32cb86bd-84ad-4a2f-a884-6f43c41b75de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:51:27.745312 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.745292 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cb86bd-84ad-4a2f-a884-6f43c41b75de-kube-api-access-2gkcr" (OuterVolumeSpecName: "kube-api-access-2gkcr") pod "32cb86bd-84ad-4a2f-a884-6f43c41b75de" (UID: "32cb86bd-84ad-4a2f-a884-6f43c41b75de"). InnerVolumeSpecName "kube-api-access-2gkcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:51:27.844190 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.844114 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gkcr\" (UniqueName: \"kubernetes.io/projected/32cb86bd-84ad-4a2f-a884-6f43c41b75de-kube-api-access-2gkcr\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:27.844190 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:27.844141 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32cb86bd-84ad-4a2f-a884-6f43c41b75de-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:28.228365 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.228261 2559 generic.go:356] "Generic (PLEG): container finished" podID="32cb86bd-84ad-4a2f-a884-6f43c41b75de" containerID="c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a" exitCode=0 Sep 11 19:51:28.228365 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.228300 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" Sep 11 19:51:28.228365 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.228343 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" event={"ID":"32cb86bd-84ad-4a2f-a884-6f43c41b75de","Type":"ContainerDied","Data":"c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a"} Sep 11 19:51:28.228643 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.228395 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw" event={"ID":"32cb86bd-84ad-4a2f-a884-6f43c41b75de","Type":"ContainerDied","Data":"79eb34a68e39773228604a78290e959f3bb41583eb88f83b3d190f58fb60dcc9"} Sep 11 19:51:28.228643 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.228418 2559 scope.go:117] "RemoveContainer" containerID="c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a" Sep 11 19:51:28.230417 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.230394 2559 generic.go:356] "Generic (PLEG): container finished" podID="4435c463-fcd7-494d-9d19-34270da006f1" containerID="423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9" exitCode=0 Sep 11 19:51:28.230564 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.230432 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" Sep 11 19:51:28.230564 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.230472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" event={"ID":"4435c463-fcd7-494d-9d19-34270da006f1","Type":"ContainerDied","Data":"423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9"} Sep 11 19:51:28.230564 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.230516 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27" event={"ID":"4435c463-fcd7-494d-9d19-34270da006f1","Type":"ContainerDied","Data":"6a5ed285c9e2b4eec956ce4d1a1229a84449a91246aa3ff38578e4adcdbac0a5"} Sep 11 19:51:28.239361 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.239253 2559 scope.go:117] "RemoveContainer" containerID="c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a" Sep 11 19:51:28.239590 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:28.239564 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a\": container with ID starting with c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a not found: ID does not exist" containerID="c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a" Sep 11 19:51:28.239708 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.239596 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a"} err="failed to get container status \"c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a\": rpc error: code = NotFound desc = could not find container \"c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a\": container with ID starting with c00b3509baadc180c689da336267ebdfbbb7458c6ac2a35ab3c8da207cde3c0a not found: ID does not exist" Sep 11 19:51:28.239708 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.239612 2559 scope.go:117] "RemoveContainer" containerID="423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9" Sep 11 19:51:28.248486 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.248466 2559 scope.go:117] "RemoveContainer" containerID="423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9" Sep 11 19:51:28.248770 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:28.248753 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9\": container with ID starting with 423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9 not found: ID does not exist" containerID="423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9" Sep 11 19:51:28.248821 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.248780 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9"} err="failed to get container status \"423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9\": rpc error: code = NotFound desc = could not find container \"423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9\": container with ID starting with 423f414ec3f662ab915bcc58887c462c88b833b22306b2f26acc8fba018107e9 not found: ID does not exist" Sep 11 19:51:28.253052 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.253031 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw"] Sep 11 19:51:28.258511 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.258486 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-872425-64d5588d54-m59bw"] Sep 11 19:51:28.269364 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.269343 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27"] Sep 11 19:51:28.272471 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:28.272451 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-872425-54d66d65f5-tjd27"] Sep 11 19:51:29.225722 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:29.225686 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25955265-4498-4038-885d-d5b6badbd241" path="/var/lib/kubelet/pods/25955265-4498-4038-885d-d5b6badbd241/volumes" Sep 11 19:51:29.226071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:29.226057 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32cb86bd-84ad-4a2f-a884-6f43c41b75de" path="/var/lib/kubelet/pods/32cb86bd-84ad-4a2f-a884-6f43c41b75de/volumes" Sep 11 19:51:29.226366 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:29.226355 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4435c463-fcd7-494d-9d19-34270da006f1" path="/var/lib/kubelet/pods/4435c463-fcd7-494d-9d19-34270da006f1/volumes" Sep 11 19:51:33.486962 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.486927 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-e36a8a"] Sep 11 19:51:33.489447 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.487332 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4435c463-fcd7-494d-9d19-34270da006f1" containerName="payload-processing" Sep 11 19:51:33.489447 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.487344 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4435c463-fcd7-494d-9d19-34270da006f1" containerName="payload-processing" Sep 11 19:51:33.489447 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.487358 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cb86bd-84ad-4a2f-a884-6f43c41b75de" containerName="payload-pre-processing" Sep 11 19:51:33.489447 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.487363 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cb86bd-84ad-4a2f-a884-6f43c41b75de" containerName="payload-pre-processing" Sep 11 19:51:33.489447 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.487423 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4435c463-fcd7-494d-9d19-34270da006f1" containerName="payload-processing" Sep 11 19:51:33.489447 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.487450 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="32cb86bd-84ad-4a2f-a884-6f43c41b75de" containerName="payload-pre-processing" Sep 11 19:51:33.490456 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.490424 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-e36a8a" Sep 11 19:51:33.493065 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.493043 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xq6qr\"" Sep 11 19:51:33.495533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.495509 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-e36a8a"] Sep 11 19:51:33.595032 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.594991 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbpcz\" (UniqueName: \"kubernetes.io/projected/102d41fb-92c6-4f6f-864a-696e33ad01d7-kube-api-access-gbpcz\") pod \"test-curl-7707-e36a8a\" (UID: \"102d41fb-92c6-4f6f-864a-696e33ad01d7\") " pod="opendatahub/test-curl-7707-e36a8a" Sep 11 19:51:33.696533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.696494 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbpcz\" (UniqueName: \"kubernetes.io/projected/102d41fb-92c6-4f6f-864a-696e33ad01d7-kube-api-access-gbpcz\") pod \"test-curl-7707-e36a8a\" (UID: \"102d41fb-92c6-4f6f-864a-696e33ad01d7\") " pod="opendatahub/test-curl-7707-e36a8a" Sep 11 19:51:33.704808 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.704770 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbpcz\" (UniqueName: \"kubernetes.io/projected/102d41fb-92c6-4f6f-864a-696e33ad01d7-kube-api-access-gbpcz\") pod \"test-curl-7707-e36a8a\" (UID: \"102d41fb-92c6-4f6f-864a-696e33ad01d7\") " pod="opendatahub/test-curl-7707-e36a8a" Sep 11 19:51:33.802699 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:33.802659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-e36a8a" Sep 11 19:51:34.102882 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:34.102844 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod102d41fb_92c6_4f6f_864a_696e33ad01d7.slice/crio-b0e9c3cecad837eae5e0b6015307f493b803f5a320a201743940c77a08bfb264 WatchSource:0}: Error finding container b0e9c3cecad837eae5e0b6015307f493b803f5a320a201743940c77a08bfb264: Status 404 returned error can't find the container with id b0e9c3cecad837eae5e0b6015307f493b803f5a320a201743940c77a08bfb264 Sep 11 19:51:34.103337 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:34.103299 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-e36a8a"] Sep 11 19:51:34.197318 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:34.197289 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:51:34.197522 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:34.197501 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:51:34.274192 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:34.274150 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-e36a8a" event={"ID":"102d41fb-92c6-4f6f-864a-696e33ad01d7","Type":"ContainerStarted","Data":"179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52"} Sep 11 19:51:34.274192 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:34.274197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-e36a8a" event={"ID":"102d41fb-92c6-4f6f-864a-696e33ad01d7","Type":"ContainerStarted","Data":"b0e9c3cecad837eae5e0b6015307f493b803f5a320a201743940c77a08bfb264"} Sep 11 19:51:34.299520 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:34.299428 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-e36a8a" podStartSLOduration=1.2994082630000001 podStartE2EDuration="1.299408263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:34.297811666 +0000 UTC m=+1527.570318868" watchObservedRunningTime="2026-09-11 19:51:34.299408263 +0000 UTC m=+1527.571915458" Sep 11 19:51:34.987461 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:34.987398 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-e36a8a"] Sep 11 19:51:34.990222 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:34.990195 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-e36a8a"] Sep 11 19:51:35.102489 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.102373 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-bdfb10"] Sep 11 19:51:35.102844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.102826 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="102d41fb-92c6-4f6f-864a-696e33ad01d7" containerName="test-curl-7707-e36a8a" Sep 11 19:51:35.102844 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.102843 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="102d41fb-92c6-4f6f-864a-696e33ad01d7" containerName="test-curl-7707-e36a8a" Sep 11 19:51:35.102996 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.102916 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="102d41fb-92c6-4f6f-864a-696e33ad01d7" containerName="test-curl-7707-e36a8a" Sep 11 19:51:35.106294 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.106274 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-bdfb10" Sep 11 19:51:35.111632 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.111601 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-bdfb10"] Sep 11 19:51:35.211673 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.211634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8fxx\" (UniqueName: \"kubernetes.io/projected/a89be7e3-dccd-43c1-ab30-af365858d54f-kube-api-access-g8fxx\") pod \"test-curl-7707-bdfb10\" (UID: \"a89be7e3-dccd-43c1-ab30-af365858d54f\") " pod="opendatahub/test-curl-7707-bdfb10" Sep 11 19:51:35.281151 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:35.281117 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-e36a8a" container="test-curl-7707-e36a8a" Sep 11 19:51:35.313237 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.313195 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8fxx\" (UniqueName: \"kubernetes.io/projected/a89be7e3-dccd-43c1-ab30-af365858d54f-kube-api-access-g8fxx\") pod \"test-curl-7707-bdfb10\" (UID: \"a89be7e3-dccd-43c1-ab30-af365858d54f\") " pod="opendatahub/test-curl-7707-bdfb10" Sep 11 19:51:35.320980 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.320946 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8fxx\" (UniqueName: \"kubernetes.io/projected/a89be7e3-dccd-43c1-ab30-af365858d54f-kube-api-access-g8fxx\") pod \"test-curl-7707-bdfb10\" (UID: \"a89be7e3-dccd-43c1-ab30-af365858d54f\") " pod="opendatahub/test-curl-7707-bdfb10" Sep 11 19:51:35.417417 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.417342 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-bdfb10" Sep 11 19:51:35.692713 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:35.692677 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-bdfb10"] Sep 11 19:51:35.697854 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:35.697812 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda89be7e3_dccd_43c1_ab30_af365858d54f.slice/crio-ad2edbe4e7935034ce054981081f37893dad74c43c330a387fb6901361ce11f4 WatchSource:0}: Error finding container ad2edbe4e7935034ce054981081f37893dad74c43c330a387fb6901361ce11f4: Status 404 returned error can't find the container with id ad2edbe4e7935034ce054981081f37893dad74c43c330a387fb6901361ce11f4 Sep 11 19:51:36.289204 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:36.289171 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-bdfb10" event={"ID":"a89be7e3-dccd-43c1-ab30-af365858d54f","Type":"ContainerStarted","Data":"6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74"} Sep 11 19:51:36.289780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:36.289209 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-e36a8a" podUID="102d41fb-92c6-4f6f-864a-696e33ad01d7" containerName="test-curl-7707-e36a8a" containerID="cri-o://179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52" gracePeriod=2 Sep 11 19:51:36.289780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:36.289215 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-bdfb10" event={"ID":"a89be7e3-dccd-43c1-ab30-af365858d54f","Type":"ContainerStarted","Data":"ad2edbe4e7935034ce054981081f37893dad74c43c330a387fb6901361ce11f4"} Sep 11 19:51:36.306609 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:36.306561 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-bdfb10" podStartSLOduration=1.306544482 podStartE2EDuration="1.306544482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:36.304854642 +0000 UTC m=+1529.577361844" watchObservedRunningTime="2026-09-11 19:51:36.306544482 +0000 UTC m=+1529.579051759" Sep 11 19:51:36.891687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:36.891631 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-bdfb10"] Sep 11 19:51:36.894723 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:36.894686 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-bdfb10"] Sep 11 19:51:37.013069 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.013036 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-bd6eb7"] Sep 11 19:51:37.013484 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.013470 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a89be7e3-dccd-43c1-ab30-af365858d54f" containerName="test-curl-7707-bdfb10" Sep 11 19:51:37.013538 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.013486 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a89be7e3-dccd-43c1-ab30-af365858d54f" containerName="test-curl-7707-bdfb10" Sep 11 19:51:37.013575 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.013556 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a89be7e3-dccd-43c1-ab30-af365858d54f" containerName="test-curl-7707-bdfb10" Sep 11 19:51:37.016670 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.016654 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-bd6eb7" Sep 11 19:51:37.023635 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.023607 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-bd6eb7"] Sep 11 19:51:37.132839 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.132803 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6grbh\" (UniqueName: \"kubernetes.io/projected/7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba-kube-api-access-6grbh\") pod \"test-curl-7707-bd6eb7\" (UID: \"7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba\") " pod="opendatahub/test-curl-7707-bd6eb7" Sep 11 19:51:37.234340 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.234310 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6grbh\" (UniqueName: \"kubernetes.io/projected/7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba-kube-api-access-6grbh\") pod \"test-curl-7707-bd6eb7\" (UID: \"7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba\") " pod="opendatahub/test-curl-7707-bd6eb7" Sep 11 19:51:37.242889 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.242864 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6grbh\" (UniqueName: \"kubernetes.io/projected/7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba-kube-api-access-6grbh\") pod \"test-curl-7707-bd6eb7\" (UID: \"7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba\") " pod="opendatahub/test-curl-7707-bd6eb7" Sep 11 19:51:37.295720 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:37.295632 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-bdfb10" container="test-curl-7707-bdfb10" Sep 11 19:51:37.328495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.328463 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-bd6eb7" Sep 11 19:51:37.606562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:37.606530 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-bd6eb7"] Sep 11 19:51:37.607269 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:37.607240 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f87b68a_f8a6_4bd6_9743_c959c6dfe1ba.slice/crio-fa8ce7c357484465ee5f9d396dfc2f4643464b7b896427a7841d5d2e1838a617 WatchSource:0}: Error finding container fa8ce7c357484465ee5f9d396dfc2f4643464b7b896427a7841d5d2e1838a617: Status 404 returned error can't find the container with id fa8ce7c357484465ee5f9d396dfc2f4643464b7b896427a7841d5d2e1838a617 Sep 11 19:51:38.304271 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.304237 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-bd6eb7" event={"ID":"7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba","Type":"ContainerStarted","Data":"5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f"} Sep 11 19:51:38.304703 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.304288 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-bd6eb7" event={"ID":"7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba","Type":"ContainerStarted","Data":"fa8ce7c357484465ee5f9d396dfc2f4643464b7b896427a7841d5d2e1838a617"} Sep 11 19:51:38.304703 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.304391 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-bdfb10" podUID="a89be7e3-dccd-43c1-ab30-af365858d54f" containerName="test-curl-7707-bdfb10" containerID="cri-o://6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74" gracePeriod=2 Sep 11 19:51:38.322257 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.322211 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-bd6eb7" podStartSLOduration=2.322196021 podStartE2EDuration="2.322196021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:38.319650117 +0000 UTC m=+1531.592157317" watchObservedRunningTime="2026-09-11 19:51:38.322196021 +0000 UTC m=+1531.594703220" Sep 11 19:51:38.447085 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.447060 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-e36a8a" Sep 11 19:51:38.544964 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.544885 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbpcz\" (UniqueName: \"kubernetes.io/projected/102d41fb-92c6-4f6f-864a-696e33ad01d7-kube-api-access-gbpcz\") pod \"102d41fb-92c6-4f6f-864a-696e33ad01d7\" (UID: \"102d41fb-92c6-4f6f-864a-696e33ad01d7\") " Sep 11 19:51:38.546971 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.546941 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/102d41fb-92c6-4f6f-864a-696e33ad01d7-kube-api-access-gbpcz" (OuterVolumeSpecName: "kube-api-access-gbpcz") pod "102d41fb-92c6-4f6f-864a-696e33ad01d7" (UID: "102d41fb-92c6-4f6f-864a-696e33ad01d7"). InnerVolumeSpecName "kube-api-access-gbpcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:51:38.646009 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.645969 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbpcz\" (UniqueName: \"kubernetes.io/projected/102d41fb-92c6-4f6f-864a-696e33ad01d7-kube-api-access-gbpcz\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:38.892384 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.892297 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-bd6eb7"] Sep 11 19:51:38.895978 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:38.895947 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-bd6eb7"] Sep 11 19:51:39.019890 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.019857 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7707-379ce1"] Sep 11 19:51:39.020289 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.020276 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba" containerName="test-curl-7707-bd6eb7" Sep 11 19:51:39.020333 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.020291 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba" containerName="test-curl-7707-bd6eb7" Sep 11 19:51:39.020381 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.020372 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba" containerName="test-curl-7707-bd6eb7" Sep 11 19:51:39.023540 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.023514 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-379ce1" Sep 11 19:51:39.034292 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.034267 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-379ce1"] Sep 11 19:51:39.051240 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.051203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lgpk\" (UniqueName: \"kubernetes.io/projected/937b4816-f583-4db8-bc28-90513c10fd4b-kube-api-access-6lgpk\") pod \"test-curl-7707-379ce1\" (UID: \"937b4816-f583-4db8-bc28-90513c10fd4b\") " pod="opendatahub/test-curl-7707-379ce1" Sep 11 19:51:39.152076 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.151995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lgpk\" (UniqueName: \"kubernetes.io/projected/937b4816-f583-4db8-bc28-90513c10fd4b-kube-api-access-6lgpk\") pod \"test-curl-7707-379ce1\" (UID: \"937b4816-f583-4db8-bc28-90513c10fd4b\") " pod="opendatahub/test-curl-7707-379ce1" Sep 11 19:51:39.161309 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.161283 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lgpk\" (UniqueName: \"kubernetes.io/projected/937b4816-f583-4db8-bc28-90513c10fd4b-kube-api-access-6lgpk\") pod \"test-curl-7707-379ce1\" (UID: \"937b4816-f583-4db8-bc28-90513c10fd4b\") " pod="opendatahub/test-curl-7707-379ce1" Sep 11 19:51:39.224136 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.224101 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="102d41fb-92c6-4f6f-864a-696e33ad01d7" path="/var/lib/kubelet/pods/102d41fb-92c6-4f6f-864a-696e33ad01d7/volumes" Sep 11 19:51:39.312277 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.312243 2559 generic.go:356] "Generic (PLEG): container finished" podID="102d41fb-92c6-4f6f-864a-696e33ad01d7" containerID="179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52" exitCode=137 Sep 11 19:51:39.312769 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.312315 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-e36a8a" Sep 11 19:51:39.312769 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.312320 2559 scope.go:117] "RemoveContainer" containerID="179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52" Sep 11 19:51:39.312769 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:39.312620 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-bd6eb7" container="test-curl-7707-bd6eb7" Sep 11 19:51:39.323202 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.323180 2559 scope.go:117] "RemoveContainer" containerID="179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52" Sep 11 19:51:39.323509 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:39.323491 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52\": container with ID starting with 179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52 not found: ID does not exist" containerID="179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52" Sep 11 19:51:39.323584 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.323522 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52"} err="failed to get container status \"179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52\": rpc error: code = NotFound desc = could not find container \"179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52\": container with ID starting with 179949b38c2dbe993f9a6d7dd4d058c92b03de10c314d7e01dcfa15146f80d52 not found: ID does not exist" Sep 11 19:51:39.332768 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.332749 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-379ce1" Sep 11 19:51:39.609857 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:39.609823 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7707-379ce1"] Sep 11 19:51:39.612581 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:39.612537 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod937b4816_f583_4db8_bc28_90513c10fd4b.slice/crio-b229555bec39fceed206434ab530fe7d2c2f8a8784ad276d74d19b5455d41d6c WatchSource:0}: Error finding container b229555bec39fceed206434ab530fe7d2c2f8a8784ad276d74d19b5455d41d6c: Status 404 returned error can't find the container with id b229555bec39fceed206434ab530fe7d2c2f8a8784ad276d74d19b5455d41d6c Sep 11 19:51:40.321795 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.321766 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-379ce1" event={"ID":"937b4816-f583-4db8-bc28-90513c10fd4b","Type":"ContainerStarted","Data":"45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4"} Sep 11 19:51:40.322154 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.321812 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7707-379ce1" event={"ID":"937b4816-f583-4db8-bc28-90513c10fd4b","Type":"ContainerStarted","Data":"b229555bec39fceed206434ab530fe7d2c2f8a8784ad276d74d19b5455d41d6c"} Sep 11 19:51:40.323058 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.323033 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-bd6eb7" podUID="7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba" containerName="test-curl-7707-bd6eb7" containerID="cri-o://5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f" gracePeriod=2 Sep 11 19:51:40.355947 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.355888 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7707-379ce1" podStartSLOduration=1.355867772 podStartE2EDuration="1.355867772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:40.354246192 +0000 UTC m=+1533.626753393" watchObservedRunningTime="2026-09-11 19:51:40.355867772 +0000 UTC m=+1533.628374970" Sep 11 19:51:40.451224 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.451197 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-bdfb10" Sep 11 19:51:40.566268 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.566174 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8fxx\" (UniqueName: \"kubernetes.io/projected/a89be7e3-dccd-43c1-ab30-af365858d54f-kube-api-access-g8fxx\") pod \"a89be7e3-dccd-43c1-ab30-af365858d54f\" (UID: \"a89be7e3-dccd-43c1-ab30-af365858d54f\") " Sep 11 19:51:40.568258 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.568225 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a89be7e3-dccd-43c1-ab30-af365858d54f-kube-api-access-g8fxx" (OuterVolumeSpecName: "kube-api-access-g8fxx") pod "a89be7e3-dccd-43c1-ab30-af365858d54f" (UID: "a89be7e3-dccd-43c1-ab30-af365858d54f"). InnerVolumeSpecName "kube-api-access-g8fxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:51:40.667924 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.667887 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8fxx\" (UniqueName: \"kubernetes.io/projected/a89be7e3-dccd-43c1-ab30-af365858d54f-kube-api-access-g8fxx\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:40.932845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.932750 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7707-379ce1"] Sep 11 19:51:40.942839 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:40.942790 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7707-379ce1"] Sep 11 19:51:41.224971 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:41.224885 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a89be7e3-dccd-43c1-ab30-af365858d54f" path="/var/lib/kubelet/pods/a89be7e3-dccd-43c1-ab30-af365858d54f/volumes" Sep 11 19:51:41.330684 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:41.330653 2559 generic.go:356] "Generic (PLEG): container finished" podID="a89be7e3-dccd-43c1-ab30-af365858d54f" containerID="6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74" exitCode=137 Sep 11 19:51:41.331104 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:41.330702 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-bdfb10" Sep 11 19:51:41.331104 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:41.330760 2559 scope.go:117] "RemoveContainer" containerID="6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74" Sep 11 19:51:41.331104 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:41.330955 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7707-379ce1" container="test-curl-7707-379ce1" Sep 11 19:51:41.343044 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:41.343020 2559 scope.go:117] "RemoveContainer" containerID="6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74" Sep 11 19:51:41.343610 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:41.343589 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74\": container with ID starting with 6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74 not found: ID does not exist" containerID="6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74" Sep 11 19:51:41.343705 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:41.343619 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74"} err="failed to get container status \"6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74\": rpc error: code = NotFound desc = could not find container \"6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74\": container with ID starting with 6945c179525bcc5b3851f33c3de230edb28e4011e213b1d7272b9e38540b9d74 not found: ID does not exist" Sep 11 19:51:42.340252 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:42.340218 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7707-379ce1" podUID="937b4816-f583-4db8-bc28-90513c10fd4b" containerName="test-curl-7707-379ce1" containerID="cri-o://45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4" gracePeriod=2 Sep 11 19:51:42.515169 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:42.515142 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-bd6eb7" Sep 11 19:51:42.588162 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:42.588074 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6grbh\" (UniqueName: \"kubernetes.io/projected/7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba-kube-api-access-6grbh\") pod \"7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba\" (UID: \"7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba\") " Sep 11 19:51:42.590262 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:42.590227 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba-kube-api-access-6grbh" (OuterVolumeSpecName: "kube-api-access-6grbh") pod "7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba" (UID: "7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba"). InnerVolumeSpecName "kube-api-access-6grbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:51:42.689143 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:42.689105 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6grbh\" (UniqueName: \"kubernetes.io/projected/7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba-kube-api-access-6grbh\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:43.017768 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.017730 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq"] Sep 11 19:51:43.018198 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.018185 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="937b4816-f583-4db8-bc28-90513c10fd4b" containerName="test-curl-7707-379ce1" Sep 11 19:51:43.018257 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.018201 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="937b4816-f583-4db8-bc28-90513c10fd4b" containerName="test-curl-7707-379ce1" Sep 11 19:51:43.018294 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.018275 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="937b4816-f583-4db8-bc28-90513c10fd4b" containerName="test-curl-7707-379ce1" Sep 11 19:51:43.021356 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.021339 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:43.024098 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.024080 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-58c78c-dockercfg-8pgc7\"" Sep 11 19:51:43.024175 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.024130 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-58c78c\"" Sep 11 19:51:43.030154 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.030128 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq"] Sep 11 19:51:43.043797 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.043305 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh"] Sep 11 19:51:43.047447 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.047415 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:43.057789 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.057765 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh"] Sep 11 19:51:43.093068 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.093039 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mpd\" (UniqueName: \"kubernetes.io/projected/f8fc3401-2495-4c9b-9a48-d2fec46767ee-kube-api-access-n9mpd\") pod \"payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq\" (UID: \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:43.093189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.093082 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcqwn\" (UniqueName: \"kubernetes.io/projected/f13784bc-b8ce-47fb-8436-ffa68726db3b-kube-api-access-dcqwn\") pod \"payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh\" (UID: \"f13784bc-b8ce-47fb-8436-ffa68726db3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:43.093189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.093115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13784bc-b8ce-47fb-8436-ffa68726db3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh\" (UID: \"f13784bc-b8ce-47fb-8436-ffa68726db3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:43.093189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.093169 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc3401-2495-4c9b-9a48-d2fec46767ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq\" (UID: \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:43.194308 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.194263 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcqwn\" (UniqueName: \"kubernetes.io/projected/f13784bc-b8ce-47fb-8436-ffa68726db3b-kube-api-access-dcqwn\") pod \"payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh\" (UID: \"f13784bc-b8ce-47fb-8436-ffa68726db3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:43.194549 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.194328 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13784bc-b8ce-47fb-8436-ffa68726db3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh\" (UID: \"f13784bc-b8ce-47fb-8436-ffa68726db3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:43.194549 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.194396 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc3401-2495-4c9b-9a48-d2fec46767ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq\" (UID: \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:43.194549 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.194525 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mpd\" (UniqueName: \"kubernetes.io/projected/f8fc3401-2495-4c9b-9a48-d2fec46767ee-kube-api-access-n9mpd\") pod \"payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq\" (UID: \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:43.195115 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.195093 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13784bc-b8ce-47fb-8436-ffa68726db3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh\" (UID: \"f13784bc-b8ce-47fb-8436-ffa68726db3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:43.195151 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.195102 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc3401-2495-4c9b-9a48-d2fec46767ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq\" (UID: \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:43.203160 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.203133 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mpd\" (UniqueName: \"kubernetes.io/projected/f8fc3401-2495-4c9b-9a48-d2fec46767ee-kube-api-access-n9mpd\") pod \"payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq\" (UID: \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:43.203351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.203329 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcqwn\" (UniqueName: \"kubernetes.io/projected/f13784bc-b8ce-47fb-8436-ffa68726db3b-kube-api-access-dcqwn\") pod \"payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh\" (UID: \"f13784bc-b8ce-47fb-8436-ffa68726db3b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:43.225884 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.225859 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba" path="/var/lib/kubelet/pods/7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba/volumes" Sep 11 19:51:43.333205 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.333114 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:43.348529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.348496 2559 scope.go:117] "RemoveContainer" containerID="5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f" Sep 11 19:51:43.349063 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.348499 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-bd6eb7" Sep 11 19:51:43.349063 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.348492 2559 generic.go:356] "Generic (PLEG): container finished" podID="7f87b68a-f8a6-4bd6-9743-c959c6dfe1ba" containerID="5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f" exitCode=137 Sep 11 19:51:43.358925 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.358897 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:43.368003 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.367976 2559 scope.go:117] "RemoveContainer" containerID="5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f" Sep 11 19:51:43.368381 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:43.368360 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f\": container with ID starting with 5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f not found: ID does not exist" containerID="5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f" Sep 11 19:51:43.368511 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.368392 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f"} err="failed to get container status \"5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f\": rpc error: code = NotFound desc = could not find container \"5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f\": container with ID starting with 5190b0ba4380e7f10a3b4dd675b095570855b3b84e3179b75f36e94df6b4ad8f not found: ID does not exist" Sep 11 19:51:43.614654 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.614626 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq"] Sep 11 19:51:43.615924 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:43.615900 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8fc3401_2495_4c9b_9a48_d2fec46767ee.slice/crio-e047beb674d5a61329db32c0fd912e42699b8c7e8e299976b12d3d7982c16e29 WatchSource:0}: Error finding container e047beb674d5a61329db32c0fd912e42699b8c7e8e299976b12d3d7982c16e29: Status 404 returned error can't find the container with id e047beb674d5a61329db32c0fd912e42699b8c7e8e299976b12d3d7982c16e29 Sep 11 19:51:43.653285 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:43.653262 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh"] Sep 11 19:51:43.654239 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:43.654214 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13784bc_b8ce_47fb_8436_ffa68726db3b.slice/crio-129d8a01500d8e89b833ba14891172346eebdb643b229effc0fa2c6e3e697401 WatchSource:0}: Error finding container 129d8a01500d8e89b833ba14891172346eebdb643b229effc0fa2c6e3e697401: Status 404 returned error can't find the container with id 129d8a01500d8e89b833ba14891172346eebdb643b229effc0fa2c6e3e697401 Sep 11 19:51:44.358742 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.358711 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" event={"ID":"f13784bc-b8ce-47fb-8436-ffa68726db3b","Type":"ContainerStarted","Data":"0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231"} Sep 11 19:51:44.359100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.358772 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" event={"ID":"f13784bc-b8ce-47fb-8436-ffa68726db3b","Type":"ContainerStarted","Data":"129d8a01500d8e89b833ba14891172346eebdb643b229effc0fa2c6e3e697401"} Sep 11 19:51:44.359100 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.358907 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:44.360655 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.360627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" event={"ID":"f8fc3401-2495-4c9b-9a48-d2fec46767ee","Type":"ContainerStarted","Data":"b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3"} Sep 11 19:51:44.360774 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.360680 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" event={"ID":"f8fc3401-2495-4c9b-9a48-d2fec46767ee","Type":"ContainerStarted","Data":"e047beb674d5a61329db32c0fd912e42699b8c7e8e299976b12d3d7982c16e29"} Sep 11 19:51:44.360774 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.360712 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:51:44.376261 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.376215 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" podStartSLOduration=1.376198539 podStartE2EDuration="1.376198539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:44.374684268 +0000 UTC m=+1537.647191470" watchObservedRunningTime="2026-09-11 19:51:44.376198539 +0000 UTC m=+1537.648705782" Sep 11 19:51:44.393972 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.393922 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" podStartSLOduration=1.393905719 podStartE2EDuration="1.393905719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:44.391194626 +0000 UTC m=+1537.663701826" watchObservedRunningTime="2026-09-11 19:51:44.393905719 +0000 UTC m=+1537.666412937" Sep 11 19:51:44.514018 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.513996 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-379ce1" Sep 11 19:51:44.611957 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.611851 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lgpk\" (UniqueName: \"kubernetes.io/projected/937b4816-f583-4db8-bc28-90513c10fd4b-kube-api-access-6lgpk\") pod \"937b4816-f583-4db8-bc28-90513c10fd4b\" (UID: \"937b4816-f583-4db8-bc28-90513c10fd4b\") " Sep 11 19:51:44.613992 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.613967 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937b4816-f583-4db8-bc28-90513c10fd4b-kube-api-access-6lgpk" (OuterVolumeSpecName: "kube-api-access-6lgpk") pod "937b4816-f583-4db8-bc28-90513c10fd4b" (UID: "937b4816-f583-4db8-bc28-90513c10fd4b"). InnerVolumeSpecName "kube-api-access-6lgpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:51:44.652898 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.652863 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:51:44.712918 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:44.712883 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lgpk\" (UniqueName: \"kubernetes.io/projected/937b4816-f583-4db8-bc28-90513c10fd4b-kube-api-access-6lgpk\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:51:45.227287 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:45.227250 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="937b4816-f583-4db8-bc28-90513c10fd4b" path="/var/lib/kubelet/pods/937b4816-f583-4db8-bc28-90513c10fd4b/volumes" Sep 11 19:51:45.372488 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:45.372454 2559 generic.go:356] "Generic (PLEG): container finished" podID="937b4816-f583-4db8-bc28-90513c10fd4b" containerID="45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4" exitCode=137 Sep 11 19:51:45.372488 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:45.372491 2559 scope.go:117] "RemoveContainer" containerID="45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4" Sep 11 19:51:45.373014 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:45.372501 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7707-379ce1" Sep 11 19:51:45.384422 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:45.384395 2559 scope.go:117] "RemoveContainer" containerID="45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4" Sep 11 19:51:45.384852 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:51:45.384825 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4\": container with ID starting with 45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4 not found: ID does not exist" containerID="45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4" Sep 11 19:51:45.384972 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:45.384863 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4"} err="failed to get container status \"45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4\": rpc error: code = NotFound desc = could not find container \"45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4\": container with ID starting with 45d3b1655f2455bd0908359ebcf8ca2a88014dd31ce6965482a84ae2bc13deb4 not found: ID does not exist" Sep 11 19:51:50.311228 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.311192 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q"] Sep 11 19:51:50.316326 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.316302 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:50.319178 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.319154 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c0517f60\"" Sep 11 19:51:50.319374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.319360 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c0517f60-dockercfg-z5t2z\"" Sep 11 19:51:50.333733 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.333708 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q"] Sep 11 19:51:50.355629 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.355598 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6"] Sep 11 19:51:50.359157 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.359139 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:50.367467 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.367427 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6"] Sep 11 19:51:50.469199 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.469157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q\" (UID: \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:50.469396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.469208 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjnc\" (UniqueName: \"kubernetes.io/projected/ecbef3f8-1943-467a-bfd4-4161fe234ab9-kube-api-access-7fjnc\") pod \"payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6\" (UID: \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\") " pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:50.469396 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.469334 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbef3f8-1943-467a-bfd4-4161fe234ab9-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6\" (UID: \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\") " pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:50.469545 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.469392 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85j48\" (UniqueName: \"kubernetes.io/projected/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-kube-api-access-85j48\") pod \"payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q\" (UID: \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:50.570989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.570858 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q\" (UID: \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:50.570989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.570937 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjnc\" (UniqueName: \"kubernetes.io/projected/ecbef3f8-1943-467a-bfd4-4161fe234ab9-kube-api-access-7fjnc\") pod \"payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6\" (UID: \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\") " pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:50.571247 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.571045 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbef3f8-1943-467a-bfd4-4161fe234ab9-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6\" (UID: \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\") " pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:50.571247 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.571099 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85j48\" (UniqueName: \"kubernetes.io/projected/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-kube-api-access-85j48\") pod \"payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q\" (UID: \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:50.571635 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.571614 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q\" (UID: \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:50.571749 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.571732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbef3f8-1943-467a-bfd4-4161fe234ab9-plugins-config-volume\") pod \"payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6\" (UID: \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\") " pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:50.579571 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.579547 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85j48\" (UniqueName: \"kubernetes.io/projected/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-kube-api-access-85j48\") pod \"payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q\" (UID: \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:50.579802 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.579780 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjnc\" (UniqueName: \"kubernetes.io/projected/ecbef3f8-1943-467a-bfd4-4161fe234ab9-kube-api-access-7fjnc\") pod \"payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6\" (UID: \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\") " pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:50.626111 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.626076 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:50.669859 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.669820 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:50.909567 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.907165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q"] Sep 11 19:51:50.962856 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:50.962827 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6"] Sep 11 19:51:50.963336 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:51:50.963256 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbef3f8_1943_467a_bfd4_4161fe234ab9.slice/crio-7f850868961b2c2f960f52d567ea66b06bcfde9b5c3b62cc0851c6e919444c64 WatchSource:0}: Error finding container 7f850868961b2c2f960f52d567ea66b06bcfde9b5c3b62cc0851c6e919444c64: Status 404 returned error can't find the container with id 7f850868961b2c2f960f52d567ea66b06bcfde9b5c3b62cc0851c6e919444c64 Sep 11 19:51:51.418938 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:51.418887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" event={"ID":"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10","Type":"ContainerStarted","Data":"45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b"} Sep 11 19:51:51.419413 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:51.418962 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:51:51.419413 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:51.418986 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" event={"ID":"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10","Type":"ContainerStarted","Data":"957af998a439ac3dde46abece5caa986b0f3221c5123d1dc573311c8a8694cd8"} Sep 11 19:51:51.420788 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:51.420760 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" event={"ID":"ecbef3f8-1943-467a-bfd4-4161fe234ab9","Type":"ContainerStarted","Data":"1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3"} Sep 11 19:51:51.420920 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:51.420806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" event={"ID":"ecbef3f8-1943-467a-bfd4-4161fe234ab9","Type":"ContainerStarted","Data":"7f850868961b2c2f960f52d567ea66b06bcfde9b5c3b62cc0851c6e919444c64"} Sep 11 19:51:51.420920 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:51.420876 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:51:51.437514 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:51.437469 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" podStartSLOduration=1.437426863 podStartE2EDuration="1.437426863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:51.433770195 +0000 UTC m=+1544.706277400" watchObservedRunningTime="2026-09-11 19:51:51.437426863 +0000 UTC m=+1544.709934063" Sep 11 19:51:51.451075 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:51.451019 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" podStartSLOduration=1.451004438 podStartE2EDuration="1.451004438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:51:51.449089359 +0000 UTC m=+1544.721596572" watchObservedRunningTime="2026-09-11 19:51:51.451004438 +0000 UTC m=+1544.723511641" Sep 11 19:51:55.375337 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:55.375308 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 19:51:55.375815 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:51:55.375521 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 19:52:02.434898 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:02.434862 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:52:02.435267 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:02.435032 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:52:07.444680 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:07.444642 2559 scope.go:117] "RemoveContainer" containerID="bdc43944908e1b3f57ce1ffa50d21be9dbf1b75e243c08063bfd6a655105f89f" Sep 11 19:52:07.454221 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:07.454197 2559 scope.go:117] "RemoveContainer" containerID="c94b4f2935f4437800346292caaf74eef92b98497bb431107154ede841fe8929" Sep 11 19:52:07.463065 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:07.463046 2559 scope.go:117] "RemoveContainer" containerID="f1403d0de3a6efdcf6d81ad863af3da3e47f47f788aa49ef3b55ef7699c48471" Sep 11 19:52:07.472023 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:07.472004 2559 scope.go:117] "RemoveContainer" containerID="56e6e7b2a9099871a54135ffbc8b7365c2fd79cf54967f2ab513af13d37395f4" Sep 11 19:52:09.561776 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:09.561747 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:52:19.526520 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.526484 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs"] Sep 11 19:52:19.530372 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.530350 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:19.533187 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.533166 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ae268e-dockercfg-lkjds\"" Sep 11 19:52:19.533838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.533821 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ae268e\"" Sep 11 19:52:19.546968 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.546940 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs"] Sep 11 19:52:19.566352 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.566327 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv"] Sep 11 19:52:19.570077 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.570059 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:19.582010 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.581988 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv"] Sep 11 19:52:19.662879 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.662843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs\" (UID: \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:19.663056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.662900 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9132a-feec-4559-821f-ebfe74cd17cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv\" (UID: \"91e9132a-feec-4559-821f-ebfe74cd17cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:19.663056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.662933 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2fr2\" (UniqueName: \"kubernetes.io/projected/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-kube-api-access-p2fr2\") pod \"payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs\" (UID: \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:19.663056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.662979 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/91e9132a-feec-4559-821f-ebfe74cd17cf-kube-api-access-l9qth\") pod \"payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv\" (UID: \"91e9132a-feec-4559-821f-ebfe74cd17cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:19.763580 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.763534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9132a-feec-4559-821f-ebfe74cd17cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv\" (UID: \"91e9132a-feec-4559-821f-ebfe74cd17cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:19.763580 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.763580 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2fr2\" (UniqueName: \"kubernetes.io/projected/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-kube-api-access-p2fr2\") pod \"payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs\" (UID: \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:19.763859 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.763629 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/91e9132a-feec-4559-821f-ebfe74cd17cf-kube-api-access-l9qth\") pod \"payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv\" (UID: \"91e9132a-feec-4559-821f-ebfe74cd17cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:19.763859 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.763752 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs\" (UID: \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:19.764531 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.764491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9132a-feec-4559-821f-ebfe74cd17cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv\" (UID: \"91e9132a-feec-4559-821f-ebfe74cd17cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:19.764531 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.764510 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs\" (UID: \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:19.772517 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.772489 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/91e9132a-feec-4559-821f-ebfe74cd17cf-kube-api-access-l9qth\") pod \"payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv\" (UID: \"91e9132a-feec-4559-821f-ebfe74cd17cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:19.772637 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.772548 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2fr2\" (UniqueName: \"kubernetes.io/projected/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-kube-api-access-p2fr2\") pod \"payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs\" (UID: \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:19.841426 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.841343 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:19.881071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:19.881038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:20.152533 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.152508 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs"] Sep 11 19:52:20.154078 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:52:20.154052 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d7e2bd_4078_4152_ba9a_5984eca3ac9d.slice/crio-e8be189070d8276ad39d55904345226a9557fde064cb283cfdeaf6ccff6995ae WatchSource:0}: Error finding container e8be189070d8276ad39d55904345226a9557fde064cb283cfdeaf6ccff6995ae: Status 404 returned error can't find the container with id e8be189070d8276ad39d55904345226a9557fde064cb283cfdeaf6ccff6995ae Sep 11 19:52:20.182547 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.182521 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv"] Sep 11 19:52:20.192686 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:52:20.192654 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91e9132a_feec_4559_821f_ebfe74cd17cf.slice/crio-d5dd47bec93f028c253f2ae575d887fd3ba897da00496285ee845c478377a6e9 WatchSource:0}: Error finding container d5dd47bec93f028c253f2ae575d887fd3ba897da00496285ee845c478377a6e9: Status 404 returned error can't find the container with id d5dd47bec93f028c253f2ae575d887fd3ba897da00496285ee845c478377a6e9 Sep 11 19:52:20.626884 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.626847 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" event={"ID":"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d","Type":"ContainerStarted","Data":"4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a"} Sep 11 19:52:20.627364 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.626913 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:52:20.627364 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.626940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" event={"ID":"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d","Type":"ContainerStarted","Data":"e8be189070d8276ad39d55904345226a9557fde064cb283cfdeaf6ccff6995ae"} Sep 11 19:52:20.628687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.628661 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" event={"ID":"91e9132a-feec-4559-821f-ebfe74cd17cf","Type":"ContainerStarted","Data":"78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd"} Sep 11 19:52:20.628833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.628703 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" event={"ID":"91e9132a-feec-4559-821f-ebfe74cd17cf","Type":"ContainerStarted","Data":"d5dd47bec93f028c253f2ae575d887fd3ba897da00496285ee845c478377a6e9"} Sep 11 19:52:20.628833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.628730 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:20.650221 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.650172 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" podStartSLOduration=1.65015854 podStartE2EDuration="1.65015854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:20.647109171 +0000 UTC m=+1573.919616371" watchObservedRunningTime="2026-09-11 19:52:20.65015854 +0000 UTC m=+1573.922665740" Sep 11 19:52:20.670123 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:20.670078 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" podStartSLOduration=1.6700640390000001 podStartE2EDuration="1.670064039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:52:20.669270159 +0000 UTC m=+1573.941777358" watchObservedRunningTime="2026-09-11 19:52:20.670064039 +0000 UTC m=+1573.942571238" Sep 11 19:52:30.157135 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:30.157093 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:52:31.636497 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:31.636466 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 19:52:31.636918 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:52:31.636585 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 19:53:07.550234 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:53:07.550193 2559 scope.go:117] "RemoveContainer" containerID="d68c3a8de3dd0e848b2df1f07a1594449012e0c567d908645c7c88f245503ee1" Sep 11 19:53:11.576104 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:53:11.576062 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:53:56.375743 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:53:56.375697 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:54:07.574300 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:07.574262 2559 scope.go:117] "RemoveContainer" containerID="19849a6b8ed86e970025c22bcd8591e4397cbf6a94f181dc5c73797e66bef5b8" Sep 11 19:54:07.583720 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:07.583700 2559 scope.go:117] "RemoveContainer" containerID="f07a0a8284e81fb9552f184a10ed98bb37731e994c0c962509976c7783fe4b04" Sep 11 19:54:29.169560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:29.169480 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:54:37.591041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.591010 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb"] Sep 11 19:54:37.594622 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.594605 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" Sep 11 19:54:37.597732 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.597712 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:54:37.597802 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.597743 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:54:37.603854 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.603819 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb"] Sep 11 19:54:37.725421 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.725381 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd4c24-8f42-4de1-a068-e863977ae7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb\" (UID: \"51dd4c24-8f42-4de1-a068-e863977ae7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" Sep 11 19:54:37.826834 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.826791 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd4c24-8f42-4de1-a068-e863977ae7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb\" (UID: \"51dd4c24-8f42-4de1-a068-e863977ae7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" Sep 11 19:54:37.827430 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.827410 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd4c24-8f42-4de1-a068-e863977ae7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb\" (UID: \"51dd4c24-8f42-4de1-a068-e863977ae7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" Sep 11 19:54:37.910234 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:37.910155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" Sep 11 19:54:38.183814 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:38.183790 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb"] Sep 11 19:54:38.185766 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:54:38.185740 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dd4c24_8f42_4de1_a068_e863977ae7f1.slice/crio-71766b6442321cd65747da1e5b4a7065273e546391f3ab3cc7ad3daab6032253 WatchSource:0}: Error finding container 71766b6442321cd65747da1e5b4a7065273e546391f3ab3cc7ad3daab6032253: Status 404 returned error can't find the container with id 71766b6442321cd65747da1e5b4a7065273e546391f3ab3cc7ad3daab6032253 Sep 11 19:54:38.543855 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:38.543825 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" event={"ID":"51dd4c24-8f42-4de1-a068-e863977ae7f1","Type":"ContainerStarted","Data":"ec24e6da8ff4964bfb040e99cc2cf0fda7b765f65b4ccaaa283be3f8f2966c1c"} Sep 11 19:54:38.543989 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:38.543912 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" event={"ID":"51dd4c24-8f42-4de1-a068-e863977ae7f1","Type":"ContainerStarted","Data":"71766b6442321cd65747da1e5b4a7065273e546391f3ab3cc7ad3daab6032253"} Sep 11 19:54:38.560041 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:38.559954 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" podStartSLOduration=1.5599377140000001 podStartE2EDuration="1.559937714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:54:38.559395865 +0000 UTC m=+1711.831903064" watchObservedRunningTime="2026-09-11 19:54:38.559937714 +0000 UTC m=+1711.832444915" Sep 11 19:54:39.551779 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:39.551749 2559 generic.go:356] "Generic (PLEG): container finished" podID="51dd4c24-8f42-4de1-a068-e863977ae7f1" containerID="ec24e6da8ff4964bfb040e99cc2cf0fda7b765f65b4ccaaa283be3f8f2966c1c" exitCode=0 Sep 11 19:54:39.552205 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:39.551834 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" event={"ID":"51dd4c24-8f42-4de1-a068-e863977ae7f1","Type":"ContainerDied","Data":"ec24e6da8ff4964bfb040e99cc2cf0fda7b765f65b4ccaaa283be3f8f2966c1c"} Sep 11 19:54:40.693412 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:40.693386 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" Sep 11 19:54:40.752319 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:40.752288 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd4c24-8f42-4de1-a068-e863977ae7f1-maas-api-service-ca\") pod \"51dd4c24-8f42-4de1-a068-e863977ae7f1\" (UID: \"51dd4c24-8f42-4de1-a068-e863977ae7f1\") " Sep 11 19:54:40.752643 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:40.752619 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51dd4c24-8f42-4de1-a068-e863977ae7f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51dd4c24-8f42-4de1-a068-e863977ae7f1" (UID: "51dd4c24-8f42-4de1-a068-e863977ae7f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:54:40.854047 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:40.853962 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51dd4c24-8f42-4de1-a068-e863977ae7f1-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:54:41.567172 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:41.567146 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" Sep 11 19:54:41.567342 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:41.567144 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb" event={"ID":"51dd4c24-8f42-4de1-a068-e863977ae7f1","Type":"ContainerDied","Data":"71766b6442321cd65747da1e5b4a7065273e546391f3ab3cc7ad3daab6032253"} Sep 11 19:54:41.567342 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:41.567248 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71766b6442321cd65747da1e5b4a7065273e546391f3ab3cc7ad3daab6032253" Sep 11 19:54:51.567874 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:51.567841 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:54:53.173634 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.173603 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6"] Sep 11 19:54:53.174023 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.173815 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" podUID="ecbef3f8-1943-467a-bfd4-4161fe234ab9" containerName="payload-processing" containerID="cri-o://1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3" gracePeriod=30 Sep 11 19:54:53.184552 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.184527 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q"] Sep 11 19:54:53.184755 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.184718 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" podUID="c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10" containerName="payload-pre-processing" containerID="cri-o://45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b" gracePeriod=30 Sep 11 19:54:53.392359 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.392251 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb"] Sep 11 19:54:53.395329 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.395304 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c0517f60-bp4rb"] Sep 11 19:54:53.474476 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.474454 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:54:53.477637 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.477618 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:54:53.576134 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.576099 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-plugins-config-volume\") pod \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\" (UID: \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\") " Sep 11 19:54:53.576303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.576160 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fjnc\" (UniqueName: \"kubernetes.io/projected/ecbef3f8-1943-467a-bfd4-4161fe234ab9-kube-api-access-7fjnc\") pod \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\" (UID: \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\") " Sep 11 19:54:53.576303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.576190 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbef3f8-1943-467a-bfd4-4161fe234ab9-plugins-config-volume\") pod \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\" (UID: \"ecbef3f8-1943-467a-bfd4-4161fe234ab9\") " Sep 11 19:54:53.576303 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.576232 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85j48\" (UniqueName: \"kubernetes.io/projected/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-kube-api-access-85j48\") pod \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\" (UID: \"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10\") " Sep 11 19:54:53.576577 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.576551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10" (UID: "c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:54:53.576640 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.576606 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbef3f8-1943-467a-bfd4-4161fe234ab9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ecbef3f8-1943-467a-bfd4-4161fe234ab9" (UID: "ecbef3f8-1943-467a-bfd4-4161fe234ab9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:54:53.578523 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.578496 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbef3f8-1943-467a-bfd4-4161fe234ab9-kube-api-access-7fjnc" (OuterVolumeSpecName: "kube-api-access-7fjnc") pod "ecbef3f8-1943-467a-bfd4-4161fe234ab9" (UID: "ecbef3f8-1943-467a-bfd4-4161fe234ab9"). InnerVolumeSpecName "kube-api-access-7fjnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:54:53.578613 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.578575 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-kube-api-access-85j48" (OuterVolumeSpecName: "kube-api-access-85j48") pod "c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10" (UID: "c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10"). InnerVolumeSpecName "kube-api-access-85j48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:54:53.654641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.654605 2559 generic.go:356] "Generic (PLEG): container finished" podID="c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10" containerID="45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b" exitCode=0 Sep 11 19:54:53.654821 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.654651 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" Sep 11 19:54:53.654821 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.654688 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" event={"ID":"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10","Type":"ContainerDied","Data":"45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b"} Sep 11 19:54:53.654821 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.654745 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q" event={"ID":"c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10","Type":"ContainerDied","Data":"957af998a439ac3dde46abece5caa986b0f3221c5123d1dc573311c8a8694cd8"} Sep 11 19:54:53.654821 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.654768 2559 scope.go:117] "RemoveContainer" containerID="45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b" Sep 11 19:54:53.656250 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.656231 2559 generic.go:356] "Generic (PLEG): container finished" podID="ecbef3f8-1943-467a-bfd4-4161fe234ab9" containerID="1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3" exitCode=0 Sep 11 19:54:53.656613 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.656273 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" Sep 11 19:54:53.656613 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.656326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" event={"ID":"ecbef3f8-1943-467a-bfd4-4161fe234ab9","Type":"ContainerDied","Data":"1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3"} Sep 11 19:54:53.656613 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.656370 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6" event={"ID":"ecbef3f8-1943-467a-bfd4-4161fe234ab9","Type":"ContainerDied","Data":"7f850868961b2c2f960f52d567ea66b06bcfde9b5c3b62cc0851c6e919444c64"} Sep 11 19:54:53.665374 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.665357 2559 scope.go:117] "RemoveContainer" containerID="45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b" Sep 11 19:54:53.665809 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:54:53.665702 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b\": container with ID starting with 45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b not found: ID does not exist" containerID="45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b" Sep 11 19:54:53.665809 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.665737 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b"} err="failed to get container status \"45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b\": rpc error: code = NotFound desc = could not find container \"45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b\": container with ID starting with 45791ee59e38cdcfc51f828f2f0dd321ccc29b7514397d32d67ae986da18141b not found: ID does not exist" Sep 11 19:54:53.665809 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.665759 2559 scope.go:117] "RemoveContainer" containerID="1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3" Sep 11 19:54:53.674991 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.674972 2559 scope.go:117] "RemoveContainer" containerID="1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3" Sep 11 19:54:53.675275 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:54:53.675256 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3\": container with ID starting with 1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3 not found: ID does not exist" containerID="1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3" Sep 11 19:54:53.675345 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.675281 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3"} err="failed to get container status \"1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3\": rpc error: code = NotFound desc = could not find container \"1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3\": container with ID starting with 1ce017ff852763309eb0cbc2b137941eabe23e7f020dfbdc0d98b975f1384dd3 not found: ID does not exist" Sep 11 19:54:53.677954 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.677929 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:54:53.677954 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.677955 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fjnc\" (UniqueName: \"kubernetes.io/projected/ecbef3f8-1943-467a-bfd4-4161fe234ab9-kube-api-access-7fjnc\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:54:53.678087 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.677964 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecbef3f8-1943-467a-bfd4-4161fe234ab9-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:54:53.678087 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.677973 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85j48\" (UniqueName: \"kubernetes.io/projected/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10-kube-api-access-85j48\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:54:53.680545 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.680524 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q"] Sep 11 19:54:53.685583 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.685556 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c0517f60-67fb4d7f4-xdh9q"] Sep 11 19:54:53.709513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.705118 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6"] Sep 11 19:54:53.709513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:53.708630 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c0517f60-579bd49545-pnxt6"] Sep 11 19:54:55.224870 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:55.224831 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51dd4c24-8f42-4de1-a068-e863977ae7f1" path="/var/lib/kubelet/pods/51dd4c24-8f42-4de1-a068-e863977ae7f1/volumes" Sep 11 19:54:55.225249 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:55.225191 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10" path="/var/lib/kubelet/pods/c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10/volumes" Sep 11 19:54:55.225530 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:54:55.225517 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecbef3f8-1943-467a-bfd4-4161fe234ab9" path="/var/lib/kubelet/pods/ecbef3f8-1943-467a-bfd4-4161fe234ab9/volumes" Sep 11 19:55:20.058132 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:55:20.058094 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:55:41.060495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:55:41.060454 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:56:07.634646 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:56:07.634613 2559 scope.go:117] "RemoveContainer" containerID="718bbd6d9802c5125d91970bd02bd4dcae4b26ea70ed338f4fc4ebcb9d9ac060" Sep 11 19:56:27.855754 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:56:27.855716 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:57:00.864347 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:00.864310 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:57:07.658070 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:07.658019 2559 scope.go:117] "RemoveContainer" containerID="8ccadf30fbd7df08c9f873585cacb75d6099522a172b1386a58e2c87927902a2" Sep 11 19:57:07.667414 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:07.667389 2559 scope.go:117] "RemoveContainer" containerID="4af4d4840daa258ab0d7e4b9f5ef16657a82b58d616e2d91b729992e712d06a1" Sep 11 19:57:29.272967 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.272890 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm"] Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273322 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10" containerName="payload-pre-processing" Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273334 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10" containerName="payload-pre-processing" Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273354 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51dd4c24-8f42-4de1-a068-e863977ae7f1" containerName="revoke-keys" Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273360 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="51dd4c24-8f42-4de1-a068-e863977ae7f1" containerName="revoke-keys" Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273377 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecbef3f8-1943-467a-bfd4-4161fe234ab9" containerName="payload-processing" Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273383 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecbef3f8-1943-467a-bfd4-4161fe234ab9" containerName="payload-processing" Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273477 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a8866e-d2ee-4c7e-aaf2-bc86f84d0e10" containerName="payload-pre-processing" Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273484 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="51dd4c24-8f42-4de1-a068-e863977ae7f1" containerName="revoke-keys" Sep 11 19:57:29.273683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.273492 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ecbef3f8-1943-467a-bfd4-4161fe234ab9" containerName="payload-processing" Sep 11 19:57:29.276675 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.276659 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:29.279666 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.279642 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d88566-dockercfg-zg7mn\"" Sep 11 19:57:29.279803 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.279642 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d88566\"" Sep 11 19:57:29.288242 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.288219 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm"] Sep 11 19:57:29.295676 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.295618 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf"] Sep 11 19:57:29.299572 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.299548 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:29.313721 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.313691 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf"] Sep 11 19:57:29.368521 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.368483 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm\" (UID: \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:29.368705 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.368557 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clskz\" (UniqueName: \"kubernetes.io/projected/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-kube-api-access-clskz\") pod \"payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm\" (UID: \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:29.469976 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.469930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm\" (UID: \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:29.470185 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.469995 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clskz\" (UniqueName: \"kubernetes.io/projected/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-kube-api-access-clskz\") pod \"payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm\" (UID: \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:29.470185 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.470084 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cthkz\" (UniqueName: \"kubernetes.io/projected/37f8bd38-752d-490d-8a82-35ab755dd742-kube-api-access-cthkz\") pod \"payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf\" (UID: \"37f8bd38-752d-490d-8a82-35ab755dd742\") " pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:29.470185 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.470112 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f8bd38-752d-490d-8a82-35ab755dd742-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf\" (UID: \"37f8bd38-752d-490d-8a82-35ab755dd742\") " pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:29.470696 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.470675 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm\" (UID: \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:29.478503 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.478483 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clskz\" (UniqueName: \"kubernetes.io/projected/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-kube-api-access-clskz\") pod \"payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm\" (UID: \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:29.571053 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.570944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cthkz\" (UniqueName: \"kubernetes.io/projected/37f8bd38-752d-490d-8a82-35ab755dd742-kube-api-access-cthkz\") pod \"payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf\" (UID: \"37f8bd38-752d-490d-8a82-35ab755dd742\") " pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:29.571053 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.570986 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f8bd38-752d-490d-8a82-35ab755dd742-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf\" (UID: \"37f8bd38-752d-490d-8a82-35ab755dd742\") " pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:29.571721 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.571699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f8bd38-752d-490d-8a82-35ab755dd742-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf\" (UID: \"37f8bd38-752d-490d-8a82-35ab755dd742\") " pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:29.580079 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.580052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cthkz\" (UniqueName: \"kubernetes.io/projected/37f8bd38-752d-490d-8a82-35ab755dd742-kube-api-access-cthkz\") pod \"payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf\" (UID: \"37f8bd38-752d-490d-8a82-35ab755dd742\") " pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:29.588893 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.588868 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:29.612454 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.612419 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:29.869550 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.869524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm"] Sep 11 19:57:29.870992 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:57:29.870963 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde0e27d7_8650_4076_ab9f_dbe7cbb8010b.slice/crio-d5d4f29ce4996858083c7cc9c62030819e2a51f70c3718a1309ca76ea6cf33d6 WatchSource:0}: Error finding container d5d4f29ce4996858083c7cc9c62030819e2a51f70c3718a1309ca76ea6cf33d6: Status 404 returned error can't find the container with id d5d4f29ce4996858083c7cc9c62030819e2a51f70c3718a1309ca76ea6cf33d6 Sep 11 19:57:29.903579 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:29.903549 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf"] Sep 11 19:57:29.907743 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:57:29.907716 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f8bd38_752d_490d_8a82_35ab755dd742.slice/crio-05f9db48de8f0768acf91920923be61ae1f088fd35e52b00ccca40330a22d609 WatchSource:0}: Error finding container 05f9db48de8f0768acf91920923be61ae1f088fd35e52b00ccca40330a22d609: Status 404 returned error can't find the container with id 05f9db48de8f0768acf91920923be61ae1f088fd35e52b00ccca40330a22d609 Sep 11 19:57:30.658953 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.658907 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:57:30.677838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.677804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" event={"ID":"37f8bd38-752d-490d-8a82-35ab755dd742","Type":"ContainerStarted","Data":"2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea"} Sep 11 19:57:30.678012 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.677864 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" event={"ID":"37f8bd38-752d-490d-8a82-35ab755dd742","Type":"ContainerStarted","Data":"05f9db48de8f0768acf91920923be61ae1f088fd35e52b00ccca40330a22d609"} Sep 11 19:57:30.678012 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.677894 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:30.679649 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.679610 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" event={"ID":"de0e27d7-8650-4076-ab9f-dbe7cbb8010b","Type":"ContainerStarted","Data":"6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79"} Sep 11 19:57:30.679763 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.679666 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:57:30.679763 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.679684 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" event={"ID":"de0e27d7-8650-4076-ab9f-dbe7cbb8010b","Type":"ContainerStarted","Data":"d5d4f29ce4996858083c7cc9c62030819e2a51f70c3718a1309ca76ea6cf33d6"} Sep 11 19:57:30.694551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.694499 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" podStartSLOduration=1.6944823599999999 podStartE2EDuration="1.69448236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:57:30.693638317 +0000 UTC m=+1883.966145516" watchObservedRunningTime="2026-09-11 19:57:30.69448236 +0000 UTC m=+1883.966989559" Sep 11 19:57:30.712748 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:30.712698 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" podStartSLOduration=1.712681562 podStartE2EDuration="1.712681562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:57:30.709726538 +0000 UTC m=+1883.982233741" watchObservedRunningTime="2026-09-11 19:57:30.712681562 +0000 UTC m=+1883.985188764" Sep 11 19:57:41.687747 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:41.687713 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:57:41.688277 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:57:41.687850 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:58:04.518937 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.518905 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b"] Sep 11 19:58:04.523038 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.523021 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:04.525996 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.525964 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6b3e57\"" Sep 11 19:58:04.526362 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.526345 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6b3e57-dockercfg-7tl2k\"" Sep 11 19:58:04.537676 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.537655 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b"] Sep 11 19:58:04.543838 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.543816 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8"] Sep 11 19:58:04.547499 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.547477 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:04.562522 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.562498 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8"] Sep 11 19:58:04.624248 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.624211 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7640cd76-de0e-4713-b2c7-4df1cb360741-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b\" (UID: \"7640cd76-de0e-4713-b2c7-4df1cb360741\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:04.624248 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.624244 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkq9\" (UniqueName: \"kubernetes.io/projected/6ac4c28e-2042-42bb-adec-e10df3e19dd5-kube-api-access-jmkq9\") pod \"payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8\" (UID: \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:04.624467 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.624279 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac4c28e-2042-42bb-adec-e10df3e19dd5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8\" (UID: \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:04.624467 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.624395 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wq2v\" (UniqueName: \"kubernetes.io/projected/7640cd76-de0e-4713-b2c7-4df1cb360741-kube-api-access-9wq2v\") pod \"payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b\" (UID: \"7640cd76-de0e-4713-b2c7-4df1cb360741\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:04.725891 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.725845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7640cd76-de0e-4713-b2c7-4df1cb360741-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b\" (UID: \"7640cd76-de0e-4713-b2c7-4df1cb360741\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:04.725891 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.725894 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkq9\" (UniqueName: \"kubernetes.io/projected/6ac4c28e-2042-42bb-adec-e10df3e19dd5-kube-api-access-jmkq9\") pod \"payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8\" (UID: \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:04.726146 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.725939 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac4c28e-2042-42bb-adec-e10df3e19dd5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8\" (UID: \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:04.726146 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.726005 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wq2v\" (UniqueName: \"kubernetes.io/projected/7640cd76-de0e-4713-b2c7-4df1cb360741-kube-api-access-9wq2v\") pod \"payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b\" (UID: \"7640cd76-de0e-4713-b2c7-4df1cb360741\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:04.726685 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.726663 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7640cd76-de0e-4713-b2c7-4df1cb360741-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b\" (UID: \"7640cd76-de0e-4713-b2c7-4df1cb360741\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:04.726754 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.726735 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac4c28e-2042-42bb-adec-e10df3e19dd5-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8\" (UID: \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:04.735001 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.734975 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wq2v\" (UniqueName: \"kubernetes.io/projected/7640cd76-de0e-4713-b2c7-4df1cb360741-kube-api-access-9wq2v\") pod \"payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b\" (UID: \"7640cd76-de0e-4713-b2c7-4df1cb360741\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:04.735766 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.735749 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkq9\" (UniqueName: \"kubernetes.io/projected/6ac4c28e-2042-42bb-adec-e10df3e19dd5-kube-api-access-jmkq9\") pod \"payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8\" (UID: \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\") " pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:04.834005 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.833917 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:04.857829 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:04.857796 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:05.115985 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.115953 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b"] Sep 11 19:58:05.116934 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:58:05.116906 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7640cd76_de0e_4713_b2c7_4df1cb360741.slice/crio-b5ab5ac899862f34e64d0072ca92d9718177614a436a441bf3f183384b901c19 WatchSource:0}: Error finding container b5ab5ac899862f34e64d0072ca92d9718177614a436a441bf3f183384b901c19: Status 404 returned error can't find the container with id b5ab5ac899862f34e64d0072ca92d9718177614a436a441bf3f183384b901c19 Sep 11 19:58:05.143203 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.143179 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8"] Sep 11 19:58:05.153537 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:58:05.153512 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac4c28e_2042_42bb_adec_e10df3e19dd5.slice/crio-47189016d69539a167e8c14efd41a4adef0405190d7b4ca3c4a88274548d1a71 WatchSource:0}: Error finding container 47189016d69539a167e8c14efd41a4adef0405190d7b4ca3c4a88274548d1a71: Status 404 returned error can't find the container with id 47189016d69539a167e8c14efd41a4adef0405190d7b4ca3c4a88274548d1a71 Sep 11 19:58:05.910482 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.910425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" event={"ID":"6ac4c28e-2042-42bb-adec-e10df3e19dd5","Type":"ContainerStarted","Data":"f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772"} Sep 11 19:58:05.910986 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.910514 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:05.910986 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.910545 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" event={"ID":"6ac4c28e-2042-42bb-adec-e10df3e19dd5","Type":"ContainerStarted","Data":"47189016d69539a167e8c14efd41a4adef0405190d7b4ca3c4a88274548d1a71"} Sep 11 19:58:05.912630 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.912600 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" event={"ID":"7640cd76-de0e-4713-b2c7-4df1cb360741","Type":"ContainerStarted","Data":"2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04"} Sep 11 19:58:05.912780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.912651 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" event={"ID":"7640cd76-de0e-4713-b2c7-4df1cb360741","Type":"ContainerStarted","Data":"b5ab5ac899862f34e64d0072ca92d9718177614a436a441bf3f183384b901c19"} Sep 11 19:58:05.912780 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.912699 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:05.926823 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.926770 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" podStartSLOduration=1.9267520500000002 podStartE2EDuration="1.92675205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:58:05.925701297 +0000 UTC m=+1919.198208501" watchObservedRunningTime="2026-09-11 19:58:05.92675205 +0000 UTC m=+1919.199259316" Sep 11 19:58:05.942659 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:05.942610 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" podStartSLOduration=1.9425955529999999 podStartE2EDuration="1.942595553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:58:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:58:05.939928841 +0000 UTC m=+1919.212436042" watchObservedRunningTime="2026-09-11 19:58:05.942595553 +0000 UTC m=+1919.215102796" Sep 11 19:58:07.692895 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:07.692863 2559 scope.go:117] "RemoveContainer" containerID="c3ae5b33599e222ea7b0d16dfc745473db56acfb5741caaf2296daa9435fa5f3" Sep 11 19:58:16.922477 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:16.922420 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:58:16.922957 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:16.922615 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:58:19.466215 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:19.466174 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:58:30.663046 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.663009 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5"] Sep 11 19:58:30.666993 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.666975 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" Sep 11 19:58:30.669731 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.669709 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:58:30.669880 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.669706 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:58:30.675633 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.675224 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5"] Sep 11 19:58:30.786837 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.786793 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884a0e1d-9a44-4377-8cbd-29c9b1494539-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d88566-gzlc5\" (UID: \"884a0e1d-9a44-4377-8cbd-29c9b1494539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" Sep 11 19:58:30.887862 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.887823 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884a0e1d-9a44-4377-8cbd-29c9b1494539-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d88566-gzlc5\" (UID: \"884a0e1d-9a44-4377-8cbd-29c9b1494539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" Sep 11 19:58:30.888601 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.888577 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884a0e1d-9a44-4377-8cbd-29c9b1494539-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d88566-gzlc5\" (UID: \"884a0e1d-9a44-4377-8cbd-29c9b1494539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" Sep 11 19:58:30.980233 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:30.980151 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" Sep 11 19:58:31.257310 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:31.257281 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5"] Sep 11 19:58:31.258585 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:58:31.258561 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884a0e1d_9a44_4377_8cbd_29c9b1494539.slice/crio-51f9b86a5483076222378566365a8440ac48179e7ad061ae7c5b5259e2ec5332 WatchSource:0}: Error finding container 51f9b86a5483076222378566365a8440ac48179e7ad061ae7c5b5259e2ec5332: Status 404 returned error can't find the container with id 51f9b86a5483076222378566365a8440ac48179e7ad061ae7c5b5259e2ec5332 Sep 11 19:58:32.083614 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:32.083519 2559 generic.go:356] "Generic (PLEG): container finished" podID="884a0e1d-9a44-4377-8cbd-29c9b1494539" containerID="431e5bbe1a3e0af83b5b31135d1250d48d329f5b39473be29d8800cb3939a01b" exitCode=0 Sep 11 19:58:32.083982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:32.083639 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" event={"ID":"884a0e1d-9a44-4377-8cbd-29c9b1494539","Type":"ContainerDied","Data":"431e5bbe1a3e0af83b5b31135d1250d48d329f5b39473be29d8800cb3939a01b"} Sep 11 19:58:32.083982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:32.083683 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" event={"ID":"884a0e1d-9a44-4377-8cbd-29c9b1494539","Type":"ContainerStarted","Data":"51f9b86a5483076222378566365a8440ac48179e7ad061ae7c5b5259e2ec5332"} Sep 11 19:58:33.243630 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:33.243601 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" Sep 11 19:58:33.315958 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:33.315918 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884a0e1d-9a44-4377-8cbd-29c9b1494539-maas-api-service-ca\") pod \"884a0e1d-9a44-4377-8cbd-29c9b1494539\" (UID: \"884a0e1d-9a44-4377-8cbd-29c9b1494539\") " Sep 11 19:58:33.316340 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:33.316318 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884a0e1d-9a44-4377-8cbd-29c9b1494539-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "884a0e1d-9a44-4377-8cbd-29c9b1494539" (UID: "884a0e1d-9a44-4377-8cbd-29c9b1494539"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:58:33.417919 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:33.417828 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/884a0e1d-9a44-4377-8cbd-29c9b1494539-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:58:34.097391 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:34.097362 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" Sep 11 19:58:34.097391 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:34.097372 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5" event={"ID":"884a0e1d-9a44-4377-8cbd-29c9b1494539","Type":"ContainerDied","Data":"51f9b86a5483076222378566365a8440ac48179e7ad061ae7c5b5259e2ec5332"} Sep 11 19:58:34.097641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:34.097417 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51f9b86a5483076222378566365a8440ac48179e7ad061ae7c5b5259e2ec5332" Sep 11 19:58:46.471701 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.471667 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf"] Sep 11 19:58:46.473037 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.471888 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" podUID="37f8bd38-752d-490d-8a82-35ab755dd742" containerName="payload-processing" containerID="cri-o://2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea" gracePeriod=30 Sep 11 19:58:46.478208 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.478179 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm"] Sep 11 19:58:46.478699 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.478671 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" podUID="de0e27d7-8650-4076-ab9f-dbe7cbb8010b" containerName="payload-pre-processing" containerID="cri-o://6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79" gracePeriod=30 Sep 11 19:58:46.647106 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.647064 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5"] Sep 11 19:58:46.650495 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.650465 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d88566-gzlc5"] Sep 11 19:58:46.764914 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.764890 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:58:46.767933 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.767911 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:58:46.847728 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.847692 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cthkz\" (UniqueName: \"kubernetes.io/projected/37f8bd38-752d-490d-8a82-35ab755dd742-kube-api-access-cthkz\") pod \"37f8bd38-752d-490d-8a82-35ab755dd742\" (UID: \"37f8bd38-752d-490d-8a82-35ab755dd742\") " Sep 11 19:58:46.847934 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.847759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f8bd38-752d-490d-8a82-35ab755dd742-plugins-config-volume\") pod \"37f8bd38-752d-490d-8a82-35ab755dd742\" (UID: \"37f8bd38-752d-490d-8a82-35ab755dd742\") " Sep 11 19:58:46.848129 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.848105 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f8bd38-752d-490d-8a82-35ab755dd742-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37f8bd38-752d-490d-8a82-35ab755dd742" (UID: "37f8bd38-752d-490d-8a82-35ab755dd742"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:58:46.850164 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.850133 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f8bd38-752d-490d-8a82-35ab755dd742-kube-api-access-cthkz" (OuterVolumeSpecName: "kube-api-access-cthkz") pod "37f8bd38-752d-490d-8a82-35ab755dd742" (UID: "37f8bd38-752d-490d-8a82-35ab755dd742"). InnerVolumeSpecName "kube-api-access-cthkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:58:46.949613 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.949577 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clskz\" (UniqueName: \"kubernetes.io/projected/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-kube-api-access-clskz\") pod \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\" (UID: \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\") " Sep 11 19:58:46.949833 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.949695 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-plugins-config-volume\") pod \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\" (UID: \"de0e27d7-8650-4076-ab9f-dbe7cbb8010b\") " Sep 11 19:58:46.949971 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.949953 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f8bd38-752d-490d-8a82-35ab755dd742-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:58:46.949971 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.949969 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cthkz\" (UniqueName: \"kubernetes.io/projected/37f8bd38-752d-490d-8a82-35ab755dd742-kube-api-access-cthkz\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:58:46.950168 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.950027 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "de0e27d7-8650-4076-ab9f-dbe7cbb8010b" (UID: "de0e27d7-8650-4076-ab9f-dbe7cbb8010b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:58:46.951945 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:46.951924 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-kube-api-access-clskz" (OuterVolumeSpecName: "kube-api-access-clskz") pod "de0e27d7-8650-4076-ab9f-dbe7cbb8010b" (UID: "de0e27d7-8650-4076-ab9f-dbe7cbb8010b"). InnerVolumeSpecName "kube-api-access-clskz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:58:47.050558 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.050473 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:58:47.050558 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.050501 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clskz\" (UniqueName: \"kubernetes.io/projected/de0e27d7-8650-4076-ab9f-dbe7cbb8010b-kube-api-access-clskz\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:58:47.185560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.185510 2559 generic.go:356] "Generic (PLEG): container finished" podID="de0e27d7-8650-4076-ab9f-dbe7cbb8010b" containerID="6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79" exitCode=0 Sep 11 19:58:47.185560 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.185553 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" Sep 11 19:58:47.185829 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.185595 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" event={"ID":"de0e27d7-8650-4076-ab9f-dbe7cbb8010b","Type":"ContainerDied","Data":"6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79"} Sep 11 19:58:47.185829 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.185647 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm" event={"ID":"de0e27d7-8650-4076-ab9f-dbe7cbb8010b","Type":"ContainerDied","Data":"d5d4f29ce4996858083c7cc9c62030819e2a51f70c3718a1309ca76ea6cf33d6"} Sep 11 19:58:47.185829 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.185673 2559 scope.go:117] "RemoveContainer" containerID="6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79" Sep 11 19:58:47.187293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.187272 2559 generic.go:356] "Generic (PLEG): container finished" podID="37f8bd38-752d-490d-8a82-35ab755dd742" containerID="2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea" exitCode=0 Sep 11 19:58:47.187400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.187326 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" Sep 11 19:58:47.187400 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.187326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" event={"ID":"37f8bd38-752d-490d-8a82-35ab755dd742","Type":"ContainerDied","Data":"2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea"} Sep 11 19:58:47.187527 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.187469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf" event={"ID":"37f8bd38-752d-490d-8a82-35ab755dd742","Type":"ContainerDied","Data":"05f9db48de8f0768acf91920923be61ae1f088fd35e52b00ccca40330a22d609"} Sep 11 19:58:47.197912 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.197896 2559 scope.go:117] "RemoveContainer" containerID="6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79" Sep 11 19:58:47.198318 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:58:47.198299 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79\": container with ID starting with 6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79 not found: ID does not exist" containerID="6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79" Sep 11 19:58:47.198386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.198329 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79"} err="failed to get container status \"6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79\": rpc error: code = NotFound desc = could not find container \"6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79\": container with ID starting with 6a4e94e5efd376d1d5ebdda485e010df511a8b7e41b1542076cc85468fd5ea79 not found: ID does not exist" Sep 11 19:58:47.198386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.198347 2559 scope.go:117] "RemoveContainer" containerID="2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea" Sep 11 19:58:47.208564 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.208542 2559 scope.go:117] "RemoveContainer" containerID="2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea" Sep 11 19:58:47.208868 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:58:47.208849 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea\": container with ID starting with 2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea not found: ID does not exist" containerID="2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea" Sep 11 19:58:47.208919 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.208875 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea"} err="failed to get container status \"2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea\": rpc error: code = NotFound desc = could not find container \"2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea\": container with ID starting with 2bd17668464c8914b01555d3303672a084380e17e16be726b4461c6747cc52ea not found: ID does not exist" Sep 11 19:58:47.213049 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.213028 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm"] Sep 11 19:58:47.224907 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.224879 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="884a0e1d-9a44-4377-8cbd-29c9b1494539" path="/var/lib/kubelet/pods/884a0e1d-9a44-4377-8cbd-29c9b1494539/volumes" Sep 11 19:58:47.225189 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.225175 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d88566-57d8d44798-x8pqm"] Sep 11 19:58:47.231426 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.231400 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf"] Sep 11 19:58:47.236633 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:47.236611 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d88566-c74d89b97-5wwmf"] Sep 11 19:58:49.224541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:49.224507 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f8bd38-752d-490d-8a82-35ab755dd742" path="/var/lib/kubelet/pods/37f8bd38-752d-490d-8a82-35ab755dd742/volumes" Sep 11 19:58:49.225237 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:49.224877 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de0e27d7-8650-4076-ab9f-dbe7cbb8010b" path="/var/lib/kubelet/pods/de0e27d7-8650-4076-ab9f-dbe7cbb8010b/volumes" Sep 11 19:58:52.955328 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:52.955286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:58:53.685027 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.684992 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj"] Sep 11 19:58:53.685525 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685508 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="884a0e1d-9a44-4377-8cbd-29c9b1494539" containerName="revoke-keys" Sep 11 19:58:53.685584 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685530 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="884a0e1d-9a44-4377-8cbd-29c9b1494539" containerName="revoke-keys" Sep 11 19:58:53.685584 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685552 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de0e27d7-8650-4076-ab9f-dbe7cbb8010b" containerName="payload-pre-processing" Sep 11 19:58:53.685584 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685558 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="de0e27d7-8650-4076-ab9f-dbe7cbb8010b" containerName="payload-pre-processing" Sep 11 19:58:53.685584 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685565 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f8bd38-752d-490d-8a82-35ab755dd742" containerName="payload-processing" Sep 11 19:58:53.685584 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685572 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f8bd38-752d-490d-8a82-35ab755dd742" containerName="payload-processing" Sep 11 19:58:53.685737 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685654 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f8bd38-752d-490d-8a82-35ab755dd742" containerName="payload-processing" Sep 11 19:58:53.685737 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685661 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="de0e27d7-8650-4076-ab9f-dbe7cbb8010b" containerName="payload-pre-processing" Sep 11 19:58:53.685737 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.685669 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="884a0e1d-9a44-4377-8cbd-29c9b1494539" containerName="revoke-keys" Sep 11 19:58:53.690334 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.690311 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" Sep 11 19:58:53.693293 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.693266 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:58:53.693653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.693628 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:58:53.696363 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.696339 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj"] Sep 11 19:58:53.816446 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.816412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72474d01-814c-4351-9426-8d6856b28a06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj\" (UID: \"72474d01-814c-4351-9426-8d6856b28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" Sep 11 19:58:53.917393 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.917355 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72474d01-814c-4351-9426-8d6856b28a06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj\" (UID: \"72474d01-814c-4351-9426-8d6856b28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" Sep 11 19:58:53.924716 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:53.918660 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72474d01-814c-4351-9426-8d6856b28a06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj\" (UID: \"72474d01-814c-4351-9426-8d6856b28a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" Sep 11 19:58:54.004498 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:54.004432 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" Sep 11 19:58:54.280052 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:54.280019 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj"] Sep 11 19:58:54.282274 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:58:54.282237 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72474d01_814c_4351_9426_8d6856b28a06.slice/crio-4c86d77708dec26df22ae8efac01a0a05d522af18db6551eabf5f172733b2339 WatchSource:0}: Error finding container 4c86d77708dec26df22ae8efac01a0a05d522af18db6551eabf5f172733b2339: Status 404 returned error can't find the container with id 4c86d77708dec26df22ae8efac01a0a05d522af18db6551eabf5f172733b2339 Sep 11 19:58:55.240551 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:55.240516 2559 generic.go:356] "Generic (PLEG): container finished" podID="72474d01-814c-4351-9426-8d6856b28a06" containerID="5f2ba40b47363d4cfc4845719a1d17dbebef7d8d61696adaa919535b680f2235" exitCode=0 Sep 11 19:58:55.241005 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:55.240581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" event={"ID":"72474d01-814c-4351-9426-8d6856b28a06","Type":"ContainerDied","Data":"5f2ba40b47363d4cfc4845719a1d17dbebef7d8d61696adaa919535b680f2235"} Sep 11 19:58:55.241005 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:55.240628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" event={"ID":"72474d01-814c-4351-9426-8d6856b28a06","Type":"ContainerStarted","Data":"4c86d77708dec26df22ae8efac01a0a05d522af18db6551eabf5f172733b2339"} Sep 11 19:58:56.398772 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:56.398745 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" Sep 11 19:58:56.543575 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:56.543482 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72474d01-814c-4351-9426-8d6856b28a06-maas-api-service-ca\") pod \"72474d01-814c-4351-9426-8d6856b28a06\" (UID: \"72474d01-814c-4351-9426-8d6856b28a06\") " Sep 11 19:58:56.543855 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:56.543833 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72474d01-814c-4351-9426-8d6856b28a06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72474d01-814c-4351-9426-8d6856b28a06" (UID: "72474d01-814c-4351-9426-8d6856b28a06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:58:56.644982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:56.644948 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72474d01-814c-4351-9426-8d6856b28a06-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:58:57.255778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:57.255752 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" Sep 11 19:58:57.255778 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:57.255756 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj" event={"ID":"72474d01-814c-4351-9426-8d6856b28a06","Type":"ContainerDied","Data":"4c86d77708dec26df22ae8efac01a0a05d522af18db6551eabf5f172733b2339"} Sep 11 19:58:57.255984 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:57.255805 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c86d77708dec26df22ae8efac01a0a05d522af18db6551eabf5f172733b2339" Sep 11 19:58:59.777009 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.776924 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq"] Sep 11 19:58:59.777422 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.777407 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72474d01-814c-4351-9426-8d6856b28a06" containerName="revoke-keys" Sep 11 19:58:59.777488 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.777425 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="72474d01-814c-4351-9426-8d6856b28a06" containerName="revoke-keys" Sep 11 19:58:59.777529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.777511 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="72474d01-814c-4351-9426-8d6856b28a06" containerName="revoke-keys" Sep 11 19:58:59.780700 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.780680 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" Sep 11 19:58:59.783644 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.783629 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:58:59.783734 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.783635 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:58:59.789543 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.789517 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq"] Sep 11 19:58:59.873457 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.873406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631f17f-e4ff-4254-b73b-c72285bffe18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq\" (UID: \"7631f17f-e4ff-4254-b73b-c72285bffe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" Sep 11 19:58:59.974881 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.974846 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631f17f-e4ff-4254-b73b-c72285bffe18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq\" (UID: \"7631f17f-e4ff-4254-b73b-c72285bffe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" Sep 11 19:58:59.975502 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:58:59.975481 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631f17f-e4ff-4254-b73b-c72285bffe18-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq\" (UID: \"7631f17f-e4ff-4254-b73b-c72285bffe18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" Sep 11 19:59:00.081259 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.081176 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v"] Sep 11 19:59:00.085076 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.085058 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" Sep 11 19:59:00.092884 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.092790 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v"] Sep 11 19:59:00.093160 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.093139 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" Sep 11 19:59:00.177351 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.177203 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c2c296-cce4-41eb-88f9-19b295488da2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v\" (UID: \"99c2c296-cce4-41eb-88f9-19b295488da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" Sep 11 19:59:00.278514 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.278472 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c2c296-cce4-41eb-88f9-19b295488da2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v\" (UID: \"99c2c296-cce4-41eb-88f9-19b295488da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" Sep 11 19:59:00.279243 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.279221 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c2c296-cce4-41eb-88f9-19b295488da2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v\" (UID: \"99c2c296-cce4-41eb-88f9-19b295488da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" Sep 11 19:59:00.383654 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.383627 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq"] Sep 11 19:59:00.384634 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:59:00.384603 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7631f17f_e4ff_4254_b73b_c72285bffe18.slice/crio-f8433dfd19205c6831920f29e1b501165b7814e3e15a5faa93b948d8e75ba9cc WatchSource:0}: Error finding container f8433dfd19205c6831920f29e1b501165b7814e3e15a5faa93b948d8e75ba9cc: Status 404 returned error can't find the container with id f8433dfd19205c6831920f29e1b501165b7814e3e15a5faa93b948d8e75ba9cc Sep 11 19:59:00.398184 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.398158 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" Sep 11 19:59:00.675210 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:00.675181 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v"] Sep 11 19:59:00.675863 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:59:00.675839 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c2c296_cce4_41eb_88f9_19b295488da2.slice/crio-2cf9e9e84ddcc27ecdb8f8241412be4098a16b3ee19c437ae10e031e9666442e WatchSource:0}: Error finding container 2cf9e9e84ddcc27ecdb8f8241412be4098a16b3ee19c437ae10e031e9666442e: Status 404 returned error can't find the container with id 2cf9e9e84ddcc27ecdb8f8241412be4098a16b3ee19c437ae10e031e9666442e Sep 11 19:59:01.283998 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:01.283903 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" event={"ID":"99c2c296-cce4-41eb-88f9-19b295488da2","Type":"ContainerStarted","Data":"2de4af8624ee862117c72b6c1dfdcb986ec954206855e7009f496a2235170e01"} Sep 11 19:59:01.283998 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:01.283963 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" event={"ID":"99c2c296-cce4-41eb-88f9-19b295488da2","Type":"ContainerStarted","Data":"2cf9e9e84ddcc27ecdb8f8241412be4098a16b3ee19c437ae10e031e9666442e"} Sep 11 19:59:01.285741 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:01.285716 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" event={"ID":"7631f17f-e4ff-4254-b73b-c72285bffe18","Type":"ContainerStarted","Data":"e3f9cd46ed7d8170f6e7af6dc4b5381bd217c352b7a0544152ae27a0c9cffd5c"} Sep 11 19:59:01.285872 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:01.285757 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" event={"ID":"7631f17f-e4ff-4254-b73b-c72285bffe18","Type":"ContainerStarted","Data":"f8433dfd19205c6831920f29e1b501165b7814e3e15a5faa93b948d8e75ba9cc"} Sep 11 19:59:01.304955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:01.304896 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" podStartSLOduration=1.304867344 podStartE2EDuration="1.304867344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:59:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:59:01.30085281 +0000 UTC m=+1974.573360012" watchObservedRunningTime="2026-09-11 19:59:01.304867344 +0000 UTC m=+1974.577374545" Sep 11 19:59:01.316375 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:01.316317 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" podStartSLOduration=2.316295053 podStartE2EDuration="2.316295053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:59:01.315809095 +0000 UTC m=+1974.588316296" watchObservedRunningTime="2026-09-11 19:59:01.316295053 +0000 UTC m=+1974.588802254" Sep 11 19:59:02.293635 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:02.293600 2559 generic.go:356] "Generic (PLEG): container finished" podID="99c2c296-cce4-41eb-88f9-19b295488da2" containerID="2de4af8624ee862117c72b6c1dfdcb986ec954206855e7009f496a2235170e01" exitCode=0 Sep 11 19:59:02.294173 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:02.293686 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" event={"ID":"99c2c296-cce4-41eb-88f9-19b295488da2","Type":"ContainerDied","Data":"2de4af8624ee862117c72b6c1dfdcb986ec954206855e7009f496a2235170e01"} Sep 11 19:59:02.295827 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:02.295801 2559 generic.go:356] "Generic (PLEG): container finished" podID="7631f17f-e4ff-4254-b73b-c72285bffe18" containerID="e3f9cd46ed7d8170f6e7af6dc4b5381bd217c352b7a0544152ae27a0c9cffd5c" exitCode=0 Sep 11 19:59:02.295955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:02.295862 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" event={"ID":"7631f17f-e4ff-4254-b73b-c72285bffe18","Type":"ContainerDied","Data":"e3f9cd46ed7d8170f6e7af6dc4b5381bd217c352b7a0544152ae27a0c9cffd5c"} Sep 11 19:59:03.464647 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:03.464619 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" Sep 11 19:59:03.467665 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:03.467647 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" Sep 11 19:59:03.511224 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:03.511189 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c2c296-cce4-41eb-88f9-19b295488da2-maas-api-service-ca\") pod \"99c2c296-cce4-41eb-88f9-19b295488da2\" (UID: \"99c2c296-cce4-41eb-88f9-19b295488da2\") " Sep 11 19:59:03.511415 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:03.511295 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631f17f-e4ff-4254-b73b-c72285bffe18-maas-api-service-ca\") pod \"7631f17f-e4ff-4254-b73b-c72285bffe18\" (UID: \"7631f17f-e4ff-4254-b73b-c72285bffe18\") " Sep 11 19:59:03.511663 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:03.511631 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c2c296-cce4-41eb-88f9-19b295488da2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "99c2c296-cce4-41eb-88f9-19b295488da2" (UID: "99c2c296-cce4-41eb-88f9-19b295488da2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:03.511663 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:03.511648 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7631f17f-e4ff-4254-b73b-c72285bffe18-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7631f17f-e4ff-4254-b73b-c72285bffe18" (UID: "7631f17f-e4ff-4254-b73b-c72285bffe18"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:03.612713 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:03.612622 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7631f17f-e4ff-4254-b73b-c72285bffe18-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:03.612713 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:03.612651 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99c2c296-cce4-41eb-88f9-19b295488da2-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:04.310971 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:04.310945 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" Sep 11 19:59:04.311149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:04.310982 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq" event={"ID":"7631f17f-e4ff-4254-b73b-c72285bffe18","Type":"ContainerDied","Data":"f8433dfd19205c6831920f29e1b501165b7814e3e15a5faa93b948d8e75ba9cc"} Sep 11 19:59:04.311149 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:04.311031 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8433dfd19205c6831920f29e1b501165b7814e3e15a5faa93b948d8e75ba9cc" Sep 11 19:59:04.312554 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:04.312528 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" Sep 11 19:59:04.312554 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:04.312542 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v" event={"ID":"99c2c296-cce4-41eb-88f9-19b295488da2","Type":"ContainerDied","Data":"2cf9e9e84ddcc27ecdb8f8241412be4098a16b3ee19c437ae10e031e9666442e"} Sep 11 19:59:04.312741 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:04.312586 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cf9e9e84ddcc27ecdb8f8241412be4098a16b3ee19c437ae10e031e9666442e" Sep 11 19:59:04.698267 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:04.698169 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj"] Sep 11 19:59:04.973473 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:04.973375 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f"] Sep 11 19:59:12.454273 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.454227 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8"] Sep 11 19:59:12.454773 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.454546 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" podUID="6ac4c28e-2042-42bb-adec-e10df3e19dd5" containerName="payload-processing" containerID="cri-o://f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772" gracePeriod=30 Sep 11 19:59:12.459346 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.459317 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b"] Sep 11 19:59:12.459777 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.459723 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" podUID="7640cd76-de0e-4713-b2c7-4df1cb360741" containerName="payload-pre-processing" containerID="cri-o://2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04" gracePeriod=30 Sep 11 19:59:12.654324 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.654293 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj"] Sep 11 19:59:12.656969 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.656945 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6b3e57-2wbzj"] Sep 11 19:59:12.737653 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.737621 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:59:12.761464 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.761420 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:59:12.802798 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.802763 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9wq2v\" (UniqueName: \"kubernetes.io/projected/7640cd76-de0e-4713-b2c7-4df1cb360741-kube-api-access-9wq2v\") pod \"7640cd76-de0e-4713-b2c7-4df1cb360741\" (UID: \"7640cd76-de0e-4713-b2c7-4df1cb360741\") " Sep 11 19:59:12.802995 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.802831 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7640cd76-de0e-4713-b2c7-4df1cb360741-plugins-config-volume\") pod \"7640cd76-de0e-4713-b2c7-4df1cb360741\" (UID: \"7640cd76-de0e-4713-b2c7-4df1cb360741\") " Sep 11 19:59:12.802995 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.802922 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac4c28e-2042-42bb-adec-e10df3e19dd5-plugins-config-volume\") pod \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\" (UID: \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\") " Sep 11 19:59:12.803123 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.803063 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmkq9\" (UniqueName: \"kubernetes.io/projected/6ac4c28e-2042-42bb-adec-e10df3e19dd5-kube-api-access-jmkq9\") pod \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\" (UID: \"6ac4c28e-2042-42bb-adec-e10df3e19dd5\") " Sep 11 19:59:12.804171 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.804139 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7640cd76-de0e-4713-b2c7-4df1cb360741-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7640cd76-de0e-4713-b2c7-4df1cb360741" (UID: "7640cd76-de0e-4713-b2c7-4df1cb360741"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:12.804891 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.804811 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac4c28e-2042-42bb-adec-e10df3e19dd5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ac4c28e-2042-42bb-adec-e10df3e19dd5" (UID: "6ac4c28e-2042-42bb-adec-e10df3e19dd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:12.806501 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.806472 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7640cd76-de0e-4713-b2c7-4df1cb360741-kube-api-access-9wq2v" (OuterVolumeSpecName: "kube-api-access-9wq2v") pod "7640cd76-de0e-4713-b2c7-4df1cb360741" (UID: "7640cd76-de0e-4713-b2c7-4df1cb360741"). InnerVolumeSpecName "kube-api-access-9wq2v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:12.806827 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.806799 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac4c28e-2042-42bb-adec-e10df3e19dd5-kube-api-access-jmkq9" (OuterVolumeSpecName: "kube-api-access-jmkq9") pod "6ac4c28e-2042-42bb-adec-e10df3e19dd5" (UID: "6ac4c28e-2042-42bb-adec-e10df3e19dd5"). InnerVolumeSpecName "kube-api-access-jmkq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:12.904513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.904477 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmkq9\" (UniqueName: \"kubernetes.io/projected/6ac4c28e-2042-42bb-adec-e10df3e19dd5-kube-api-access-jmkq9\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:12.904513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.904507 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wq2v\" (UniqueName: \"kubernetes.io/projected/7640cd76-de0e-4713-b2c7-4df1cb360741-kube-api-access-9wq2v\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:12.904513 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.904520 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7640cd76-de0e-4713-b2c7-4df1cb360741-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:12.904743 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:12.904528 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ac4c28e-2042-42bb-adec-e10df3e19dd5-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:13.225453 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.225401 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72474d01-814c-4351-9426-8d6856b28a06" path="/var/lib/kubelet/pods/72474d01-814c-4351-9426-8d6856b28a06/volumes" Sep 11 19:59:13.370753 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.370721 2559 generic.go:356] "Generic (PLEG): container finished" podID="6ac4c28e-2042-42bb-adec-e10df3e19dd5" containerID="f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772" exitCode=0 Sep 11 19:59:13.370949 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.370764 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" Sep 11 19:59:13.370949 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.370803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" event={"ID":"6ac4c28e-2042-42bb-adec-e10df3e19dd5","Type":"ContainerDied","Data":"f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772"} Sep 11 19:59:13.370949 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.370856 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8" event={"ID":"6ac4c28e-2042-42bb-adec-e10df3e19dd5","Type":"ContainerDied","Data":"47189016d69539a167e8c14efd41a4adef0405190d7b4ca3c4a88274548d1a71"} Sep 11 19:59:13.370949 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.370871 2559 scope.go:117] "RemoveContainer" containerID="f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772" Sep 11 19:59:13.372395 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.372370 2559 generic.go:356] "Generic (PLEG): container finished" podID="7640cd76-de0e-4713-b2c7-4df1cb360741" containerID="2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04" exitCode=0 Sep 11 19:59:13.372562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.372406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" event={"ID":"7640cd76-de0e-4713-b2c7-4df1cb360741","Type":"ContainerDied","Data":"2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04"} Sep 11 19:59:13.372562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.372481 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" event={"ID":"7640cd76-de0e-4713-b2c7-4df1cb360741","Type":"ContainerDied","Data":"b5ab5ac899862f34e64d0072ca92d9718177614a436a441bf3f183384b901c19"} Sep 11 19:59:13.372562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.372413 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b" Sep 11 19:59:13.381107 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.381092 2559 scope.go:117] "RemoveContainer" containerID="f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772" Sep 11 19:59:13.381363 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:13.381346 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772\": container with ID starting with f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772 not found: ID does not exist" containerID="f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772" Sep 11 19:59:13.381414 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.381372 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772"} err="failed to get container status \"f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772\": rpc error: code = NotFound desc = could not find container \"f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772\": container with ID starting with f1e51965ad70757d5d4d4700b193b8f5f42c2a0810b59c716605dddd19a4f772 not found: ID does not exist" Sep 11 19:59:13.381414 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.381389 2559 scope.go:117] "RemoveContainer" containerID="2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04" Sep 11 19:59:13.390176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.390163 2559 scope.go:117] "RemoveContainer" containerID="2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04" Sep 11 19:59:13.390480 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:13.390431 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04\": container with ID starting with 2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04 not found: ID does not exist" containerID="2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04" Sep 11 19:59:13.390573 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.390479 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04"} err="failed to get container status \"2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04\": rpc error: code = NotFound desc = could not find container \"2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04\": container with ID starting with 2450c8e522862e74cc7478722863f09d1cea281d70633e40059600bc91e5fd04 not found: ID does not exist" Sep 11 19:59:13.393711 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.393691 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b"] Sep 11 19:59:13.397321 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.397302 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6b3e57-85bb6dfbbb-bpf7b"] Sep 11 19:59:13.411518 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.411491 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8"] Sep 11 19:59:13.417921 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.417898 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6b3e57-74b47b5ccc-znvd8"] Sep 11 19:59:13.738114 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.738084 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m"] Sep 11 19:59:13.738529 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.738303 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" podUID="8b5285cd-e61b-4620-ac4f-b60e0372ef6c" containerName="payload-processing" containerID="cri-o://6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197" gracePeriod=30 Sep 11 19:59:13.747829 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.747803 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h"] Sep 11 19:59:13.748053 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:13.748030 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" podUID="23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd" containerName="payload-pre-processing" containerID="cri-o://80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be" gracePeriod=30 Sep 11 19:59:14.027109 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.027081 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:59:14.032687 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.032666 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:59:14.089369 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.089337 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v"] Sep 11 19:59:14.093920 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.093894 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c7766c-xsp8v"] Sep 11 19:59:14.119466 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.119411 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-plugins-config-volume\") pod \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\" (UID: \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\") " Sep 11 19:59:14.119618 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.119518 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wl6dg\" (UniqueName: \"kubernetes.io/projected/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-kube-api-access-wl6dg\") pod \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\" (UID: \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\") " Sep 11 19:59:14.119618 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.119589 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-plugins-config-volume\") pod \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\" (UID: \"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd\") " Sep 11 19:59:14.119756 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.119628 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ff99\" (UniqueName: \"kubernetes.io/projected/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-kube-api-access-2ff99\") pod \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\" (UID: \"8b5285cd-e61b-4620-ac4f-b60e0372ef6c\") " Sep 11 19:59:14.119839 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.119804 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8b5285cd-e61b-4620-ac4f-b60e0372ef6c" (UID: "8b5285cd-e61b-4620-ac4f-b60e0372ef6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:14.120012 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.119981 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd" (UID: "23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:14.120143 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.120010 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:14.121817 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.121793 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-kube-api-access-wl6dg" (OuterVolumeSpecName: "kube-api-access-wl6dg") pod "23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd" (UID: "23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd"). InnerVolumeSpecName "kube-api-access-wl6dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:14.121917 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.121900 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-kube-api-access-2ff99" (OuterVolumeSpecName: "kube-api-access-2ff99") pod "8b5285cd-e61b-4620-ac4f-b60e0372ef6c" (UID: "8b5285cd-e61b-4620-ac4f-b60e0372ef6c"). InnerVolumeSpecName "kube-api-access-2ff99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:14.221196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.221158 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:14.221196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.221189 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ff99\" (UniqueName: \"kubernetes.io/projected/8b5285cd-e61b-4620-ac4f-b60e0372ef6c-kube-api-access-2ff99\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:14.221196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.221197 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl6dg\" (UniqueName: \"kubernetes.io/projected/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd-kube-api-access-wl6dg\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:14.386329 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.386234 2559 generic.go:356] "Generic (PLEG): container finished" podID="8b5285cd-e61b-4620-ac4f-b60e0372ef6c" containerID="6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197" exitCode=0 Sep 11 19:59:14.386329 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.386294 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" Sep 11 19:59:14.386607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.386318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" event={"ID":"8b5285cd-e61b-4620-ac4f-b60e0372ef6c","Type":"ContainerDied","Data":"6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197"} Sep 11 19:59:14.386607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.386412 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m" event={"ID":"8b5285cd-e61b-4620-ac4f-b60e0372ef6c","Type":"ContainerDied","Data":"fa91e2c8b8150eb15b1c9aaf9762a127ffe48012c371060de46c498a941ae5c6"} Sep 11 19:59:14.386607 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.386482 2559 scope.go:117] "RemoveContainer" containerID="6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197" Sep 11 19:59:14.387961 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.387937 2559 generic.go:356] "Generic (PLEG): container finished" podID="23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd" containerID="80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be" exitCode=0 Sep 11 19:59:14.388117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.388019 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" Sep 11 19:59:14.388117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.388061 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" event={"ID":"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd","Type":"ContainerDied","Data":"80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be"} Sep 11 19:59:14.388117 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.388110 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h" event={"ID":"23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd","Type":"ContainerDied","Data":"b5083ff0567e8106fb8cc319d25c249d14d70b54cef8c48c8e75db95c1a32af7"} Sep 11 19:59:14.397998 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.397842 2559 scope.go:117] "RemoveContainer" containerID="6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197" Sep 11 19:59:14.398121 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:14.398102 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197\": container with ID starting with 6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197 not found: ID does not exist" containerID="6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197" Sep 11 19:59:14.398172 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.398130 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197"} err="failed to get container status \"6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197\": rpc error: code = NotFound desc = could not find container \"6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197\": container with ID starting with 6454a9a2d8ed04b7f3b56918ad1773cc42d017ccf689d46e60a2864438997197 not found: ID does not exist" Sep 11 19:59:14.398172 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.398147 2559 scope.go:117] "RemoveContainer" containerID="80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be" Sep 11 19:59:14.408201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.408171 2559 scope.go:117] "RemoveContainer" containerID="80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be" Sep 11 19:59:14.408518 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:14.408491 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be\": container with ID starting with 80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be not found: ID does not exist" containerID="80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be" Sep 11 19:59:14.408579 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.408530 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be"} err="failed to get container status \"80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be\": rpc error: code = NotFound desc = could not find container \"80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be\": container with ID starting with 80fc46d24f7c5593d1b841d9b5bba1b5bc511ec3986e1c22f1f42c45d836c1be not found: ID does not exist" Sep 11 19:59:14.414955 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.414932 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m"] Sep 11 19:59:14.421150 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.421131 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-c7766c-85c4d88b55-kq24m"] Sep 11 19:59:14.433119 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.433097 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h"] Sep 11 19:59:14.436197 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:14.436177 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-c7766c-7f4f9fbdc8-9mj2h"] Sep 11 19:59:15.225231 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:15.225196 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd" path="/var/lib/kubelet/pods/23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd/volumes" Sep 11 19:59:15.225628 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:15.225586 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ac4c28e-2042-42bb-adec-e10df3e19dd5" path="/var/lib/kubelet/pods/6ac4c28e-2042-42bb-adec-e10df3e19dd5/volumes" Sep 11 19:59:15.225912 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:15.225898 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7640cd76-de0e-4713-b2c7-4df1cb360741" path="/var/lib/kubelet/pods/7640cd76-de0e-4713-b2c7-4df1cb360741/volumes" Sep 11 19:59:15.226218 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:15.226207 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b5285cd-e61b-4620-ac4f-b60e0372ef6c" path="/var/lib/kubelet/pods/8b5285cd-e61b-4620-ac4f-b60e0372ef6c/volumes" Sep 11 19:59:15.226825 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:15.226810 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99c2c296-cce4-41eb-88f9-19b295488da2" path="/var/lib/kubelet/pods/99c2c296-cce4-41eb-88f9-19b295488da2/volumes" Sep 11 19:59:17.574662 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:17.574628 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq"] Sep 11 19:59:17.582343 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:17.582309 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-10cc24-qqnnq"] Sep 11 19:59:19.228925 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:19.228889 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7631f17f-e4ff-4254-b73b-c72285bffe18" path="/var/lib/kubelet/pods/7631f17f-e4ff-4254-b73b-c72285bffe18/volumes" Sep 11 19:59:19.719276 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:19.719236 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" podUID="9f977182-c120-4088-ba89-e300e3d347d0" containerName="main" containerID="cri-o://dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b" gracePeriod=45 Sep 11 19:59:19.775047 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:19.775012 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f977182_c120_4088_ba89_e300e3d347d0.slice/crio-conmon-dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b.scope\": RecentStats: unable to find data in memory cache]" Sep 11 19:59:19.993982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:19.993940 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" podUID="8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" containerName="main" containerID="cri-o://f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7" gracePeriod=45 Sep 11 19:59:20.000658 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.000631 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:59:20.075515 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.075486 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-dshm\") pod \"9f977182-c120-4088-ba89-e300e3d347d0\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " Sep 11 19:59:20.075700 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.075525 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f977182-c120-4088-ba89-e300e3d347d0-tls-certs\") pod \"9f977182-c120-4088-ba89-e300e3d347d0\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " Sep 11 19:59:20.075700 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.075584 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npq8v\" (UniqueName: \"kubernetes.io/projected/9f977182-c120-4088-ba89-e300e3d347d0-kube-api-access-npq8v\") pod \"9f977182-c120-4088-ba89-e300e3d347d0\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " Sep 11 19:59:20.075700 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.075613 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-tmp-dir\") pod \"9f977182-c120-4088-ba89-e300e3d347d0\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " Sep 11 19:59:20.075700 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.075635 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-model-cache\") pod \"9f977182-c120-4088-ba89-e300e3d347d0\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " Sep 11 19:59:20.075910 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.075731 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-home\") pod \"9f977182-c120-4088-ba89-e300e3d347d0\" (UID: \"9f977182-c120-4088-ba89-e300e3d347d0\") " Sep 11 19:59:20.075990 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.075954 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f977182-c120-4088-ba89-e300e3d347d0" (UID: "9f977182-c120-4088-ba89-e300e3d347d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:59:20.076095 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.076064 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-home" (OuterVolumeSpecName: "home") pod "9f977182-c120-4088-ba89-e300e3d347d0" (UID: "9f977182-c120-4088-ba89-e300e3d347d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:59:20.076228 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.076116 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f977182-c120-4088-ba89-e300e3d347d0" (UID: "9f977182-c120-4088-ba89-e300e3d347d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:59:20.078084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.078054 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-dshm" (OuterVolumeSpecName: "dshm") pod "9f977182-c120-4088-ba89-e300e3d347d0" (UID: "9f977182-c120-4088-ba89-e300e3d347d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:59:20.078184 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.078125 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f977182-c120-4088-ba89-e300e3d347d0-kube-api-access-npq8v" (OuterVolumeSpecName: "kube-api-access-npq8v") pod "9f977182-c120-4088-ba89-e300e3d347d0" (UID: "9f977182-c120-4088-ba89-e300e3d347d0"). InnerVolumeSpecName "kube-api-access-npq8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:20.078256 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.078239 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f977182-c120-4088-ba89-e300e3d347d0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f977182-c120-4088-ba89-e300e3d347d0" (UID: "9f977182-c120-4088-ba89-e300e3d347d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:59:20.177099 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.177062 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npq8v\" (UniqueName: \"kubernetes.io/projected/9f977182-c120-4088-ba89-e300e3d347d0-kube-api-access-npq8v\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.177099 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.177096 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-tmp-dir\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.177336 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.177109 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-model-cache\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.177336 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.177118 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-home\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.177336 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.177126 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f977182-c120-4088-ba89-e300e3d347d0-dshm\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.177336 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.177133 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f977182-c120-4088-ba89-e300e3d347d0-tls-certs\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.247350 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.247320 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:59:20.301558 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.301519 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph"] Sep 11 19:59:20.302065 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302046 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ac4c28e-2042-42bb-adec-e10df3e19dd5" containerName="payload-processing" Sep 11 19:59:20.302065 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302067 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ac4c28e-2042-42bb-adec-e10df3e19dd5" containerName="payload-processing" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302090 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b5285cd-e61b-4620-ac4f-b60e0372ef6c" containerName="payload-processing" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302098 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b5285cd-e61b-4620-ac4f-b60e0372ef6c" containerName="payload-processing" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302113 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd" containerName="payload-pre-processing" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302122 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd" containerName="payload-pre-processing" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302132 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" containerName="main" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302140 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" containerName="main" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302150 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7640cd76-de0e-4713-b2c7-4df1cb360741" containerName="payload-pre-processing" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302158 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7640cd76-de0e-4713-b2c7-4df1cb360741" containerName="payload-pre-processing" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302168 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99c2c296-cce4-41eb-88f9-19b295488da2" containerName="revoke-keys" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302177 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="99c2c296-cce4-41eb-88f9-19b295488da2" containerName="revoke-keys" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302194 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7631f17f-e4ff-4254-b73b-c72285bffe18" containerName="revoke-keys" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302203 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7631f17f-e4ff-4254-b73b-c72285bffe18" containerName="revoke-keys" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302213 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f977182-c120-4088-ba89-e300e3d347d0" containerName="main" Sep 11 19:59:20.302305 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302221 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f977182-c120-4088-ba89-e300e3d347d0" containerName="main" Sep 11 19:59:20.303008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302340 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7640cd76-de0e-4713-b2c7-4df1cb360741" containerName="payload-pre-processing" Sep 11 19:59:20.303008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302352 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f977182-c120-4088-ba89-e300e3d347d0" containerName="main" Sep 11 19:59:20.303008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302363 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ac4c28e-2042-42bb-adec-e10df3e19dd5" containerName="payload-processing" Sep 11 19:59:20.303008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302373 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ec1ef0-b03b-4a04-abf1-3fa38c6de0cd" containerName="payload-pre-processing" Sep 11 19:59:20.303008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302384 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="99c2c296-cce4-41eb-88f9-19b295488da2" containerName="revoke-keys" Sep 11 19:59:20.303008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302391 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b5285cd-e61b-4620-ac4f-b60e0372ef6c" containerName="payload-processing" Sep 11 19:59:20.303008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302402 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7631f17f-e4ff-4254-b73b-c72285bffe18" containerName="revoke-keys" Sep 11 19:59:20.303008 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.302413 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" containerName="main" Sep 11 19:59:20.305991 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.305957 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" Sep 11 19:59:20.308786 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.308764 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:59:20.309145 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.309104 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:59:20.314071 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.314047 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph"] Sep 11 19:59:20.379004 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.378976 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-home\") pod \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " Sep 11 19:59:20.379176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379017 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-dshm\") pod \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " Sep 11 19:59:20.379176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379078 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-model-cache\") pod \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " Sep 11 19:59:20.379176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379137 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tmp-dir\") pod \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " Sep 11 19:59:20.379176 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379173 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzv6f\" (UniqueName: \"kubernetes.io/projected/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-kube-api-access-jzv6f\") pod \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " Sep 11 19:59:20.379401 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379203 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tls-certs\") pod \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\" (UID: \"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129\") " Sep 11 19:59:20.379401 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379306 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-home" (OuterVolumeSpecName: "home") pod "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" (UID: "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:59:20.379401 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379357 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" (UID: "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:59:20.379599 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" (UID: "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:59:20.379599 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379499 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcbcd4-96e1-47ff-b240-b12423d5409c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph\" (UID: \"7dfcbcd4-96e1-47ff-b240-b12423d5409c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" Sep 11 19:59:20.379718 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379660 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tmp-dir\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.379718 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379677 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-home\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.379718 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.379686 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-model-cache\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.381465 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.381413 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-dshm" (OuterVolumeSpecName: "dshm") pod "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" (UID: "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 19:59:20.381578 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.381478 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-kube-api-access-jzv6f" (OuterVolumeSpecName: "kube-api-access-jzv6f") pod "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" (UID: "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129"). InnerVolumeSpecName "kube-api-access-jzv6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:20.381578 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.381512 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" (UID: "8ec30a3e-46f7-4a6a-9d25-824a9bcbb129"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 19:59:20.438570 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.438540 2559 generic.go:356] "Generic (PLEG): container finished" podID="8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" containerID="f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7" exitCode=0 Sep 11 19:59:20.438769 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.438592 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" Sep 11 19:59:20.438769 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.438622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" event={"ID":"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129","Type":"ContainerDied","Data":"f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7"} Sep 11 19:59:20.438769 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.438663 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f" event={"ID":"8ec30a3e-46f7-4a6a-9d25-824a9bcbb129","Type":"ContainerDied","Data":"48ca4b82f056d33e57fb2b5746328c9d04cea8c53ef6dc7ff3dfeb1567d22f1c"} Sep 11 19:59:20.438769 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.438685 2559 scope.go:117] "RemoveContainer" containerID="f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7" Sep 11 19:59:20.440204 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.440178 2559 generic.go:356] "Generic (PLEG): container finished" podID="9f977182-c120-4088-ba89-e300e3d347d0" containerID="dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b" exitCode=0 Sep 11 19:59:20.440520 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.440222 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" Sep 11 19:59:20.440520 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.440277 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" event={"ID":"9f977182-c120-4088-ba89-e300e3d347d0","Type":"ContainerDied","Data":"dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b"} Sep 11 19:59:20.440520 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.440318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj" event={"ID":"9f977182-c120-4088-ba89-e300e3d347d0","Type":"ContainerDied","Data":"8f5eb9d6dba0a2b5f7bf2e14d79549e339d4cc0e61a0e985f480975871c287a4"} Sep 11 19:59:20.449056 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.449013 2559 scope.go:117] "RemoveContainer" containerID="f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7" Sep 11 19:59:20.449430 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:20.449394 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7\": container with ID starting with f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7 not found: ID does not exist" containerID="f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7" Sep 11 19:59:20.449543 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.449431 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7"} err="failed to get container status \"f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7\": rpc error: code = NotFound desc = could not find container \"f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7\": container with ID starting with f511b9a0ddd64490a590c89f30c88333b1b5e4aae7287831f71a5595bb1c0cb7 not found: ID does not exist" Sep 11 19:59:20.449543 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.449483 2559 scope.go:117] "RemoveContainer" containerID="dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b" Sep 11 19:59:20.458836 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.458815 2559 scope.go:117] "RemoveContainer" containerID="dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b" Sep 11 19:59:20.459143 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:20.459127 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b\": container with ID starting with dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b not found: ID does not exist" containerID="dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b" Sep 11 19:59:20.459196 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.459150 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b"} err="failed to get container status \"dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b\": rpc error: code = NotFound desc = could not find container \"dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b\": container with ID starting with dbb17ac407148964bdecd6ca7308d4242001cb5edec818ee0f38a8b61432184b not found: ID does not exist" Sep 11 19:59:20.465869 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.465829 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f"] Sep 11 19:59:20.477683 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.477657 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c7766c/simulator-w4-c7766c-kserve-7db674b865-lxf2f"] Sep 11 19:59:20.480159 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.480138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcbcd4-96e1-47ff-b240-b12423d5409c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph\" (UID: \"7dfcbcd4-96e1-47ff-b240-b12423d5409c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" Sep 11 19:59:20.480273 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.480218 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzv6f\" (UniqueName: \"kubernetes.io/projected/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-kube-api-access-jzv6f\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.480273 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.480229 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-tls-certs\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.480273 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.480239 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129-dshm\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:20.480850 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.480826 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcbcd4-96e1-47ff-b240-b12423d5409c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph\" (UID: \"7dfcbcd4-96e1-47ff-b240-b12423d5409c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" Sep 11 19:59:20.489732 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.489676 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj"] Sep 11 19:59:20.493387 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.493368 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10cc24/simulator-w0-10cc24-kserve-5fd7ff46c4-9ndlj"] Sep 11 19:59:20.621302 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.621271 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" Sep 11 19:59:20.896138 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:20.896115 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph"] Sep 11 19:59:20.898184 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:59:20.898155 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dfcbcd4_96e1_47ff_b240_b12423d5409c.slice/crio-c485356c3c218000ca2491e6fbc2251e377d39b6026c390321ac10c066a62c02 WatchSource:0}: Error finding container c485356c3c218000ca2491e6fbc2251e377d39b6026c390321ac10c066a62c02: Status 404 returned error can't find the container with id c485356c3c218000ca2491e6fbc2251e377d39b6026c390321ac10c066a62c02 Sep 11 19:59:21.226280 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:21.226249 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ec30a3e-46f7-4a6a-9d25-824a9bcbb129" path="/var/lib/kubelet/pods/8ec30a3e-46f7-4a6a-9d25-824a9bcbb129/volumes" Sep 11 19:59:21.226691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:21.226677 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f977182-c120-4088-ba89-e300e3d347d0" path="/var/lib/kubelet/pods/9f977182-c120-4088-ba89-e300e3d347d0/volumes" Sep 11 19:59:21.451641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:21.451605 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" event={"ID":"7dfcbcd4-96e1-47ff-b240-b12423d5409c","Type":"ContainerStarted","Data":"5d3575ee648900b09c88ff99a34412613bdfbad914d552b8d28193041dc87800"} Sep 11 19:59:21.452113 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:21.451660 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" event={"ID":"7dfcbcd4-96e1-47ff-b240-b12423d5409c","Type":"ContainerStarted","Data":"c485356c3c218000ca2491e6fbc2251e377d39b6026c390321ac10c066a62c02"} Sep 11 19:59:21.473320 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:21.473188 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" podStartSLOduration=1.473165138 podStartE2EDuration="1.473165138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:59:21.46693288 +0000 UTC m=+1994.739440079" watchObservedRunningTime="2026-09-11 19:59:21.473165138 +0000 UTC m=+1994.745672340" Sep 11 19:59:22.459919 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:22.459883 2559 generic.go:356] "Generic (PLEG): container finished" podID="7dfcbcd4-96e1-47ff-b240-b12423d5409c" containerID="5d3575ee648900b09c88ff99a34412613bdfbad914d552b8d28193041dc87800" exitCode=0 Sep 11 19:59:22.460308 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:22.459930 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" event={"ID":"7dfcbcd4-96e1-47ff-b240-b12423d5409c","Type":"ContainerDied","Data":"5d3575ee648900b09c88ff99a34412613bdfbad914d552b8d28193041dc87800"} Sep 11 19:59:23.605208 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:23.605168 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" Sep 11 19:59:23.713530 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:23.713499 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcbcd4-96e1-47ff-b240-b12423d5409c-maas-api-service-ca\") pod \"7dfcbcd4-96e1-47ff-b240-b12423d5409c\" (UID: \"7dfcbcd4-96e1-47ff-b240-b12423d5409c\") " Sep 11 19:59:23.713915 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:23.713888 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dfcbcd4-96e1-47ff-b240-b12423d5409c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7dfcbcd4-96e1-47ff-b240-b12423d5409c" (UID: "7dfcbcd4-96e1-47ff-b240-b12423d5409c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:23.815255 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:23.815214 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7dfcbcd4-96e1-47ff-b240-b12423d5409c-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:24.474179 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:24.474146 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" Sep 11 19:59:24.474357 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:24.474182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph" event={"ID":"7dfcbcd4-96e1-47ff-b240-b12423d5409c","Type":"ContainerDied","Data":"c485356c3c218000ca2491e6fbc2251e377d39b6026c390321ac10c066a62c02"} Sep 11 19:59:24.474357 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:24.474233 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c485356c3c218000ca2491e6fbc2251e377d39b6026c390321ac10c066a62c02" Sep 11 19:59:31.037911 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.037873 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc"] Sep 11 19:59:31.038314 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.038097 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" podUID="371f23d0-9226-4fcd-b670-2f54b406e3b5" containerName="payload-processing" containerID="cri-o://366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d" gracePeriod=30 Sep 11 19:59:31.048306 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.048272 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb"] Sep 11 19:59:31.048695 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.048620 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" podUID="fa1e6743-3084-43c1-b4c9-ba2f6ae09012" containerName="payload-pre-processing" containerID="cri-o://4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240" gracePeriod=30 Sep 11 19:59:31.206673 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.206607 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph"] Sep 11 19:59:31.209753 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.209722 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d14d7e-d5mph"] Sep 11 19:59:31.228012 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.227981 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7dfcbcd4-96e1-47ff-b240-b12423d5409c" path="/var/lib/kubelet/pods/7dfcbcd4-96e1-47ff-b240-b12423d5409c/volumes" Sep 11 19:59:31.336791 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.336768 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:59:31.339069 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.339051 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:59:31.490675 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.490639 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ztxtq\" (UniqueName: \"kubernetes.io/projected/371f23d0-9226-4fcd-b670-2f54b406e3b5-kube-api-access-ztxtq\") pod \"371f23d0-9226-4fcd-b670-2f54b406e3b5\" (UID: \"371f23d0-9226-4fcd-b670-2f54b406e3b5\") " Sep 11 19:59:31.490675 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.490672 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78kkw\" (UniqueName: \"kubernetes.io/projected/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-kube-api-access-78kkw\") pod \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\" (UID: \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\") " Sep 11 19:59:31.490901 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.490770 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371f23d0-9226-4fcd-b670-2f54b406e3b5-plugins-config-volume\") pod \"371f23d0-9226-4fcd-b670-2f54b406e3b5\" (UID: \"371f23d0-9226-4fcd-b670-2f54b406e3b5\") " Sep 11 19:59:31.490901 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.490821 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-plugins-config-volume\") pod \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\" (UID: \"fa1e6743-3084-43c1-b4c9-ba2f6ae09012\") " Sep 11 19:59:31.491206 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.491178 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371f23d0-9226-4fcd-b670-2f54b406e3b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "371f23d0-9226-4fcd-b670-2f54b406e3b5" (UID: "371f23d0-9226-4fcd-b670-2f54b406e3b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:31.491206 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.491197 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa1e6743-3084-43c1-b4c9-ba2f6ae09012" (UID: "fa1e6743-3084-43c1-b4c9-ba2f6ae09012"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:31.492864 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.492842 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371f23d0-9226-4fcd-b670-2f54b406e3b5-kube-api-access-ztxtq" (OuterVolumeSpecName: "kube-api-access-ztxtq") pod "371f23d0-9226-4fcd-b670-2f54b406e3b5" (UID: "371f23d0-9226-4fcd-b670-2f54b406e3b5"). InnerVolumeSpecName "kube-api-access-ztxtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:31.492969 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.492949 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-kube-api-access-78kkw" (OuterVolumeSpecName: "kube-api-access-78kkw") pod "fa1e6743-3084-43c1-b4c9-ba2f6ae09012" (UID: "fa1e6743-3084-43c1-b4c9-ba2f6ae09012"). InnerVolumeSpecName "kube-api-access-78kkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:31.521825 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.521794 2559 generic.go:356] "Generic (PLEG): container finished" podID="371f23d0-9226-4fcd-b670-2f54b406e3b5" containerID="366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d" exitCode=0 Sep 11 19:59:31.521960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.521837 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" Sep 11 19:59:31.521960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.521887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" event={"ID":"371f23d0-9226-4fcd-b670-2f54b406e3b5","Type":"ContainerDied","Data":"366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d"} Sep 11 19:59:31.521960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.521943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc" event={"ID":"371f23d0-9226-4fcd-b670-2f54b406e3b5","Type":"ContainerDied","Data":"3c60c9bbcc0d7e246e5ee057c9d148921e90c9999f608cbf48483995f0177612"} Sep 11 19:59:31.522076 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.521971 2559 scope.go:117] "RemoveContainer" containerID="366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d" Sep 11 19:59:31.523583 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.523560 2559 generic.go:356] "Generic (PLEG): container finished" podID="fa1e6743-3084-43c1-b4c9-ba2f6ae09012" containerID="4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240" exitCode=0 Sep 11 19:59:31.523686 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.523592 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" event={"ID":"fa1e6743-3084-43c1-b4c9-ba2f6ae09012","Type":"ContainerDied","Data":"4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240"} Sep 11 19:59:31.523686 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.523606 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" Sep 11 19:59:31.523686 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.523633 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb" event={"ID":"fa1e6743-3084-43c1-b4c9-ba2f6ae09012","Type":"ContainerDied","Data":"6cf83739e8e7c306fc9f9495d2a9d86e1537f926d428fde0bc7f1df0e7c88a5c"} Sep 11 19:59:31.532729 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.532707 2559 scope.go:117] "RemoveContainer" containerID="366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d" Sep 11 19:59:31.533038 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:31.533013 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d\": container with ID starting with 366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d not found: ID does not exist" containerID="366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d" Sep 11 19:59:31.533101 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.533052 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d"} err="failed to get container status \"366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d\": rpc error: code = NotFound desc = could not find container \"366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d\": container with ID starting with 366d9abc71161d4eb26cac512011698de3505b17b0ade2a4480a907a15e23a0d not found: ID does not exist" Sep 11 19:59:31.533101 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.533072 2559 scope.go:117] "RemoveContainer" containerID="4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240" Sep 11 19:59:31.541673 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.541657 2559 scope.go:117] "RemoveContainer" containerID="4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240" Sep 11 19:59:31.541973 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:31.541953 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240\": container with ID starting with 4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240 not found: ID does not exist" containerID="4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240" Sep 11 19:59:31.542084 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.541978 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240"} err="failed to get container status \"4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240\": rpc error: code = NotFound desc = could not find container \"4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240\": container with ID starting with 4fd4aa7b1b6dc3e3547526c75df43508153db77f9f940bcf95702dad0419a240 not found: ID does not exist" Sep 11 19:59:31.545830 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.545805 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc"] Sep 11 19:59:31.548678 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.548658 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d14d7e-54799f7779-w9glc"] Sep 11 19:59:31.562654 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.562596 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb"] Sep 11 19:59:31.567998 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.567979 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d14d7e-789f4d9854-lznzb"] Sep 11 19:59:31.592208 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.592176 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztxtq\" (UniqueName: \"kubernetes.io/projected/371f23d0-9226-4fcd-b670-2f54b406e3b5-kube-api-access-ztxtq\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:31.592208 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.592205 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78kkw\" (UniqueName: \"kubernetes.io/projected/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-kube-api-access-78kkw\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:31.592353 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.592215 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/371f23d0-9226-4fcd-b670-2f54b406e3b5-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:31.592353 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.592223 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa1e6743-3084-43c1-b4c9-ba2f6ae09012-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:31.769803 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:31.769761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:59:33.229745 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:33.229708 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="371f23d0-9226-4fcd-b670-2f54b406e3b5" path="/var/lib/kubelet/pods/371f23d0-9226-4fcd-b670-2f54b406e3b5/volumes" Sep 11 19:59:33.230946 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:33.230907 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa1e6743-3084-43c1-b4c9-ba2f6ae09012" path="/var/lib/kubelet/pods/fa1e6743-3084-43c1-b4c9-ba2f6ae09012/volumes" Sep 11 19:59:37.981452 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981405 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw"] Sep 11 19:59:37.981845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981831 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="371f23d0-9226-4fcd-b670-2f54b406e3b5" containerName="payload-processing" Sep 11 19:59:37.981845 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981846 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="371f23d0-9226-4fcd-b670-2f54b406e3b5" containerName="payload-processing" Sep 11 19:59:37.981916 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981860 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa1e6743-3084-43c1-b4c9-ba2f6ae09012" containerName="payload-pre-processing" Sep 11 19:59:37.981916 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981866 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa1e6743-3084-43c1-b4c9-ba2f6ae09012" containerName="payload-pre-processing" Sep 11 19:59:37.981916 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981880 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7dfcbcd4-96e1-47ff-b240-b12423d5409c" containerName="revoke-keys" Sep 11 19:59:37.981916 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981887 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7dfcbcd4-96e1-47ff-b240-b12423d5409c" containerName="revoke-keys" Sep 11 19:59:37.982031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981943 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7dfcbcd4-96e1-47ff-b240-b12423d5409c" containerName="revoke-keys" Sep 11 19:59:37.982031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981953 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa1e6743-3084-43c1-b4c9-ba2f6ae09012" containerName="payload-pre-processing" Sep 11 19:59:37.982031 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.981960 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="371f23d0-9226-4fcd-b670-2f54b406e3b5" containerName="payload-processing" Sep 11 19:59:37.986201 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.986186 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" Sep 11 19:59:37.989559 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.989528 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 19:59:37.990781 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:37.990752 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 19:59:38.006673 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:38.006636 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw"] Sep 11 19:59:38.153982 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:38.153941 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f8c25d-3c02-4cea-aa81-33dda68229fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw\" (UID: \"a6f8c25d-3c02-4cea-aa81-33dda68229fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" Sep 11 19:59:38.254900 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:38.254856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f8c25d-3c02-4cea-aa81-33dda68229fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw\" (UID: \"a6f8c25d-3c02-4cea-aa81-33dda68229fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" Sep 11 19:59:38.255541 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:38.255519 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f8c25d-3c02-4cea-aa81-33dda68229fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw\" (UID: \"a6f8c25d-3c02-4cea-aa81-33dda68229fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" Sep 11 19:59:38.309915 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:38.309877 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" Sep 11 19:59:38.583810 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:38.583786 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw"] Sep 11 19:59:38.585664 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:59:38.585634 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f8c25d_3c02_4cea_aa81_33dda68229fc.slice/crio-200ed072014565d4ecd095d80025b0414602787324046a6eb6c1b6334b9c99c1 WatchSource:0}: Error finding container 200ed072014565d4ecd095d80025b0414602787324046a6eb6c1b6334b9c99c1: Status 404 returned error can't find the container with id 200ed072014565d4ecd095d80025b0414602787324046a6eb6c1b6334b9c99c1 Sep 11 19:59:39.577562 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:39.577530 2559 generic.go:356] "Generic (PLEG): container finished" podID="a6f8c25d-3c02-4cea-aa81-33dda68229fc" containerID="d40c8251ef290375b39e3005150b748788c595f8aea97238b005db97fe8d0ec3" exitCode=0 Sep 11 19:59:39.577960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:39.577621 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" event={"ID":"a6f8c25d-3c02-4cea-aa81-33dda68229fc","Type":"ContainerDied","Data":"d40c8251ef290375b39e3005150b748788c595f8aea97238b005db97fe8d0ec3"} Sep 11 19:59:39.577960 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:39.577676 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" event={"ID":"a6f8c25d-3c02-4cea-aa81-33dda68229fc","Type":"ContainerStarted","Data":"200ed072014565d4ecd095d80025b0414602787324046a6eb6c1b6334b9c99c1"} Sep 11 19:59:40.725095 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:40.725070 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" Sep 11 19:59:40.777409 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:40.777378 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f8c25d-3c02-4cea-aa81-33dda68229fc-maas-api-service-ca\") pod \"a6f8c25d-3c02-4cea-aa81-33dda68229fc\" (UID: \"a6f8c25d-3c02-4cea-aa81-33dda68229fc\") " Sep 11 19:59:40.777729 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:40.777708 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f8c25d-3c02-4cea-aa81-33dda68229fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a6f8c25d-3c02-4cea-aa81-33dda68229fc" (UID: "a6f8c25d-3c02-4cea-aa81-33dda68229fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:40.878219 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:40.878133 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6f8c25d-3c02-4cea-aa81-33dda68229fc-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:41.591507 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:41.591470 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" event={"ID":"a6f8c25d-3c02-4cea-aa81-33dda68229fc","Type":"ContainerDied","Data":"200ed072014565d4ecd095d80025b0414602787324046a6eb6c1b6334b9c99c1"} Sep 11 19:59:41.591507 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:41.591488 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw" Sep 11 19:59:41.591717 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:41.591520 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="200ed072014565d4ecd095d80025b0414602787324046a6eb6c1b6334b9c99c1" Sep 11 19:59:48.774886 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:48.774857 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w"] Sep 11 19:59:48.775285 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:48.775064 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" podUID="e6c566ba-c83a-4133-93f3-838605ce413b" containerName="payload-processing" containerID="cri-o://67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1" gracePeriod=30 Sep 11 19:59:48.787925 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:48.787896 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l"] Sep 11 19:59:48.788130 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:48.788110 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" podUID="17d66807-fd8e-42e0-801c-cb27842c9007" containerName="payload-pre-processing" containerID="cri-o://1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33" gracePeriod=30 Sep 11 19:59:48.996865 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:48.996834 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw"] Sep 11 19:59:48.999977 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:48.999856 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-58f57d-nsrnw"] Sep 11 19:59:49.045705 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.045641 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:59:49.071075 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.071044 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:59:49.155058 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.155021 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m2jld\" (UniqueName: \"kubernetes.io/projected/17d66807-fd8e-42e0-801c-cb27842c9007-kube-api-access-m2jld\") pod \"17d66807-fd8e-42e0-801c-cb27842c9007\" (UID: \"17d66807-fd8e-42e0-801c-cb27842c9007\") " Sep 11 19:59:49.155251 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.155074 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c566ba-c83a-4133-93f3-838605ce413b-plugins-config-volume\") pod \"e6c566ba-c83a-4133-93f3-838605ce413b\" (UID: \"e6c566ba-c83a-4133-93f3-838605ce413b\") " Sep 11 19:59:49.155251 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.155146 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27cxj\" (UniqueName: \"kubernetes.io/projected/e6c566ba-c83a-4133-93f3-838605ce413b-kube-api-access-27cxj\") pod \"e6c566ba-c83a-4133-93f3-838605ce413b\" (UID: \"e6c566ba-c83a-4133-93f3-838605ce413b\") " Sep 11 19:59:49.155251 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.155223 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d66807-fd8e-42e0-801c-cb27842c9007-plugins-config-volume\") pod \"17d66807-fd8e-42e0-801c-cb27842c9007\" (UID: \"17d66807-fd8e-42e0-801c-cb27842c9007\") " Sep 11 19:59:49.155461 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.155413 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6c566ba-c83a-4133-93f3-838605ce413b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6c566ba-c83a-4133-93f3-838605ce413b" (UID: "e6c566ba-c83a-4133-93f3-838605ce413b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:49.155572 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.155552 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c566ba-c83a-4133-93f3-838605ce413b-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:49.155635 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.155621 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d66807-fd8e-42e0-801c-cb27842c9007-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "17d66807-fd8e-42e0-801c-cb27842c9007" (UID: "17d66807-fd8e-42e0-801c-cb27842c9007"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 19:59:49.157478 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.157430 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d66807-fd8e-42e0-801c-cb27842c9007-kube-api-access-m2jld" (OuterVolumeSpecName: "kube-api-access-m2jld") pod "17d66807-fd8e-42e0-801c-cb27842c9007" (UID: "17d66807-fd8e-42e0-801c-cb27842c9007"). InnerVolumeSpecName "kube-api-access-m2jld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:49.157578 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.157478 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6c566ba-c83a-4133-93f3-838605ce413b-kube-api-access-27cxj" (OuterVolumeSpecName: "kube-api-access-27cxj") pod "e6c566ba-c83a-4133-93f3-838605ce413b" (UID: "e6c566ba-c83a-4133-93f3-838605ce413b"). InnerVolumeSpecName "kube-api-access-27cxj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 19:59:49.230919 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.230875 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f8c25d-3c02-4cea-aa81-33dda68229fc" path="/var/lib/kubelet/pods/a6f8c25d-3c02-4cea-aa81-33dda68229fc/volumes" Sep 11 19:59:49.256983 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.256956 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/17d66807-fd8e-42e0-801c-cb27842c9007-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:49.256983 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.256980 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2jld\" (UniqueName: \"kubernetes.io/projected/17d66807-fd8e-42e0-801c-cb27842c9007-kube-api-access-m2jld\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:49.257112 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.256989 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27cxj\" (UniqueName: \"kubernetes.io/projected/e6c566ba-c83a-4133-93f3-838605ce413b-kube-api-access-27cxj\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 19:59:49.650894 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.650862 2559 generic.go:356] "Generic (PLEG): container finished" podID="17d66807-fd8e-42e0-801c-cb27842c9007" containerID="1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33" exitCode=0 Sep 11 19:59:49.651083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.650902 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" event={"ID":"17d66807-fd8e-42e0-801c-cb27842c9007","Type":"ContainerDied","Data":"1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33"} Sep 11 19:59:49.651083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.650934 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" Sep 11 19:59:49.651083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.650969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l" event={"ID":"17d66807-fd8e-42e0-801c-cb27842c9007","Type":"ContainerDied","Data":"648827aa28c116344e360c5e04042aa92bbecd9d0c33c65dda517bcdb18bc326"} Sep 11 19:59:49.651083 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.650994 2559 scope.go:117] "RemoveContainer" containerID="1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33" Sep 11 19:59:49.652572 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.652552 2559 generic.go:356] "Generic (PLEG): container finished" podID="e6c566ba-c83a-4133-93f3-838605ce413b" containerID="67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1" exitCode=0 Sep 11 19:59:49.652688 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.652616 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" Sep 11 19:59:49.652688 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.652647 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" event={"ID":"e6c566ba-c83a-4133-93f3-838605ce413b","Type":"ContainerDied","Data":"67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1"} Sep 11 19:59:49.652795 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.652703 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w" event={"ID":"e6c566ba-c83a-4133-93f3-838605ce413b","Type":"ContainerDied","Data":"e7e13c9654207ff9593e1b59b8d6d374f86f82e2c20ab721597b43dc3bba872d"} Sep 11 19:59:49.661298 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.661282 2559 scope.go:117] "RemoveContainer" containerID="1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33" Sep 11 19:59:49.661580 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:49.661562 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33\": container with ID starting with 1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33 not found: ID does not exist" containerID="1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33" Sep 11 19:59:49.661641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.661587 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33"} err="failed to get container status \"1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33\": rpc error: code = NotFound desc = could not find container \"1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33\": container with ID starting with 1b3aec048b5dec7e40a570ec0bf635a7250ea95715d29e241dea2559042f6a33 not found: ID does not exist" Sep 11 19:59:49.661641 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.661602 2559 scope.go:117] "RemoveContainer" containerID="67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1" Sep 11 19:59:49.670581 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.670547 2559 scope.go:117] "RemoveContainer" containerID="67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1" Sep 11 19:59:49.670858 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:49.670835 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1\": container with ID starting with 67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1 not found: ID does not exist" containerID="67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1" Sep 11 19:59:49.670942 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.670865 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1"} err="failed to get container status \"67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1\": rpc error: code = NotFound desc = could not find container \"67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1\": container with ID starting with 67b03fa4cb7662c4b2b1a319220c01846970004766f1200caf4875ee495ad3d1 not found: ID does not exist" Sep 11 19:59:49.673411 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.673386 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l"] Sep 11 19:59:49.678182 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.678160 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-58f57d-749b5d4cf4-wkl5l"] Sep 11 19:59:49.692150 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.692120 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w"] Sep 11 19:59:49.699186 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:49.699158 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-58f57d-6fddfc6565-k8k9w"] Sep 11 19:59:51.223933 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:51.223902 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17d66807-fd8e-42e0-801c-cb27842c9007" path="/var/lib/kubelet/pods/17d66807-fd8e-42e0-801c-cb27842c9007/volumes" Sep 11 19:59:51.224386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:51.224261 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6c566ba-c83a-4133-93f3-838605ce413b" path="/var/lib/kubelet/pods/e6c566ba-c83a-4133-93f3-838605ce413b/volumes" Sep 11 19:59:54.852553 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:54.852518 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 19:59:57.091078 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091041 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7710-5a0309"] Sep 11 19:59:57.091491 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091420 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17d66807-fd8e-42e0-801c-cb27842c9007" containerName="payload-pre-processing" Sep 11 19:59:57.091491 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091431 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="17d66807-fd8e-42e0-801c-cb27842c9007" containerName="payload-pre-processing" Sep 11 19:59:57.091491 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091456 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6c566ba-c83a-4133-93f3-838605ce413b" containerName="payload-processing" Sep 11 19:59:57.091491 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091461 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6c566ba-c83a-4133-93f3-838605ce413b" containerName="payload-processing" Sep 11 19:59:57.091491 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091482 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f8c25d-3c02-4cea-aa81-33dda68229fc" containerName="revoke-keys" Sep 11 19:59:57.091491 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091488 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f8c25d-3c02-4cea-aa81-33dda68229fc" containerName="revoke-keys" Sep 11 19:59:57.091691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091552 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6c566ba-c83a-4133-93f3-838605ce413b" containerName="payload-processing" Sep 11 19:59:57.091691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091563 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="17d66807-fd8e-42e0-801c-cb27842c9007" containerName="payload-pre-processing" Sep 11 19:59:57.091691 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.091571 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f8c25d-3c02-4cea-aa81-33dda68229fc" containerName="revoke-keys" Sep 11 19:59:57.096195 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.096155 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-5a0309" Sep 11 19:59:57.099004 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.098981 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xq6qr\"" Sep 11 19:59:57.099258 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.099105 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-5a0309"] Sep 11 19:59:57.231708 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.231677 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdms8\" (UniqueName: \"kubernetes.io/projected/53615f32-af7a-48ce-9e18-ee4d6788730b-kube-api-access-hdms8\") pod \"test-curl-7710-5a0309\" (UID: \"53615f32-af7a-48ce-9e18-ee4d6788730b\") " pod="opendatahub/test-curl-7710-5a0309" Sep 11 19:59:57.332753 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.332719 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdms8\" (UniqueName: \"kubernetes.io/projected/53615f32-af7a-48ce-9e18-ee4d6788730b-kube-api-access-hdms8\") pod \"test-curl-7710-5a0309\" (UID: \"53615f32-af7a-48ce-9e18-ee4d6788730b\") " pod="opendatahub/test-curl-7710-5a0309" Sep 11 19:59:57.341038 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.341004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdms8\" (UniqueName: \"kubernetes.io/projected/53615f32-af7a-48ce-9e18-ee4d6788730b-kube-api-access-hdms8\") pod \"test-curl-7710-5a0309\" (UID: \"53615f32-af7a-48ce-9e18-ee4d6788730b\") " pod="opendatahub/test-curl-7710-5a0309" Sep 11 19:59:57.408285 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.408205 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-5a0309" Sep 11 19:59:57.684184 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.684157 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-5a0309"] Sep 11 19:59:57.686236 ip-10-0-137-97 kubenswrapper[2559]: W0911 19:59:57.686209 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53615f32_af7a_48ce_9e18_ee4d6788730b.slice/crio-fa9c70332f9e8fbadb5b0a190e7cd4c8d3f883b3041e42ca686a942061618170 WatchSource:0}: Error finding container fa9c70332f9e8fbadb5b0a190e7cd4c8d3f883b3041e42ca686a942061618170: Status 404 returned error can't find the container with id fa9c70332f9e8fbadb5b0a190e7cd4c8d3f883b3041e42ca686a942061618170 Sep 11 19:59:57.713553 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:57.713525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-5a0309" event={"ID":"53615f32-af7a-48ce-9e18-ee4d6788730b","Type":"ContainerStarted","Data":"fa9c70332f9e8fbadb5b0a190e7cd4c8d3f883b3041e42ca686a942061618170"} Sep 11 19:59:58.721292 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:58.721257 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-5a0309" event={"ID":"53615f32-af7a-48ce-9e18-ee4d6788730b","Type":"ContainerStarted","Data":"2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5"} Sep 11 19:59:58.738213 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:58.738150 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7710-5a0309" podStartSLOduration=1.738135776 podStartE2EDuration="1.738135776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 19:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 19:59:58.736179936 +0000 UTC m=+2032.008687136" watchObservedRunningTime="2026-09-11 19:59:58.738135776 +0000 UTC m=+2032.010642976" Sep 11 19:59:59.324232 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:59.324192 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7710-5a0309"] Sep 11 19:59:59.328386 ip-10-0-137-97 kubenswrapper[2559]: I0911 19:59:59.328350 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7710-5a0309"] Sep 11 19:59:59.727636 ip-10-0-137-97 kubenswrapper[2559]: E0911 19:59:59.727603 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7710-5a0309" container="test-curl-7710-5a0309" Sep 11 20:00:00.133724 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.133686 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh"] Sep 11 20:00:00.134140 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.134124 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53615f32-af7a-48ce-9e18-ee4d6788730b" containerName="test-curl-7710-5a0309" Sep 11 20:00:00.134192 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.134141 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="53615f32-af7a-48ce-9e18-ee4d6788730b" containerName="test-curl-7710-5a0309" Sep 11 20:00:00.134232 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.134217 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="53615f32-af7a-48ce-9e18-ee4d6788730b" containerName="test-curl-7710-5a0309" Sep 11 20:00:00.137573 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.137546 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" Sep 11 20:00:00.139510 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.139485 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc"] Sep 11 20:00:00.140818 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.140797 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 20:00:00.142904 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.142882 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" Sep 11 20:00:00.148622 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.148604 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" Sep 11 20:00:00.153107 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.152884 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh"] Sep 11 20:00:00.157044 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.157011 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc"] Sep 11 20:00:00.157843 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.157818 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" Sep 11 20:00:00.430990 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.430957 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh"] Sep 11 20:00:00.432889 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:00:00.432859 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f9a898_14fe_4d54_9ddf_597d6b0743b3.slice/crio-d76d6290035c585cd16e7713828c9516361dc8afdda01a0922c8dd059b34a24a WatchSource:0}: Error finding container d76d6290035c585cd16e7713828c9516361dc8afdda01a0922c8dd059b34a24a: Status 404 returned error can't find the container with id d76d6290035c585cd16e7713828c9516361dc8afdda01a0922c8dd059b34a24a Sep 11 20:00:00.456544 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.456519 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc"] Sep 11 20:00:00.456860 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:00:00.456832 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42f74184_da9b_4932_9603_b46696237ae7.slice/crio-ba2400d788034c8c5e6e190db376f9544794167950abbdd9d04a6da5dad9c417 WatchSource:0}: Error finding container ba2400d788034c8c5e6e190db376f9544794167950abbdd9d04a6da5dad9c417: Status 404 returned error can't find the container with id ba2400d788034c8c5e6e190db376f9544794167950abbdd9d04a6da5dad9c417 Sep 11 20:00:00.736849 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.736819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" event={"ID":"42f74184-da9b-4932-9603-b46696237ae7","Type":"ContainerStarted","Data":"cb59003a09b4be3663507ac4f793fae2b9db76c39f3e002e8d149d5c88501dc1"} Sep 11 20:00:00.737243 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.736892 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" event={"ID":"42f74184-da9b-4932-9603-b46696237ae7","Type":"ContainerStarted","Data":"ba2400d788034c8c5e6e190db376f9544794167950abbdd9d04a6da5dad9c417"} Sep 11 20:00:00.738730 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.738701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" event={"ID":"d0f9a898-14fe-4d54-9ddf-597d6b0743b3","Type":"ContainerStarted","Data":"deb1a36e1b6c8c0baff85efb47da616a5394342da673679463e33a0d1c2e5958"} Sep 11 20:00:00.738823 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.738753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" event={"ID":"d0f9a898-14fe-4d54-9ddf-597d6b0743b3","Type":"ContainerStarted","Data":"d76d6290035c585cd16e7713828c9516361dc8afdda01a0922c8dd059b34a24a"} Sep 11 20:00:00.738823 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.738735 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7710-5a0309" podUID="53615f32-af7a-48ce-9e18-ee4d6788730b" containerName="test-curl-7710-5a0309" containerID="cri-o://2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5" gracePeriod=2 Sep 11 20:00:00.751008 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.750954 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" podStartSLOduration=0.750937253 podStartE2EDuration="750.937253ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:00.75003303 +0000 UTC m=+2034.022540229" watchObservedRunningTime="2026-09-11 20:00:00.750937253 +0000 UTC m=+2034.023444452" Sep 11 20:00:00.752088 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:00:00.752060 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7710-5a0309\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="53615f32-af7a-48ce-9e18-ee4d6788730b" pod="opendatahub/test-curl-7710-5a0309" Sep 11 20:00:00.769967 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:00.769920 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" podStartSLOduration=0.76990444 podStartE2EDuration="769.90444ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:00:00.76763358 +0000 UTC m=+2034.040140781" watchObservedRunningTime="2026-09-11 20:00:00.76990444 +0000 UTC m=+2034.042411640" Sep 11 20:00:01.746758 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:01.746721 2559 generic.go:356] "Generic (PLEG): container finished" podID="42f74184-da9b-4932-9603-b46696237ae7" containerID="cb59003a09b4be3663507ac4f793fae2b9db76c39f3e002e8d149d5c88501dc1" exitCode=0 Sep 11 20:00:01.747196 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:01.746819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" event={"ID":"42f74184-da9b-4932-9603-b46696237ae7","Type":"ContainerDied","Data":"cb59003a09b4be3663507ac4f793fae2b9db76c39f3e002e8d149d5c88501dc1"} Sep 11 20:00:01.748491 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:01.748467 2559 generic.go:356] "Generic (PLEG): container finished" podID="d0f9a898-14fe-4d54-9ddf-597d6b0743b3" containerID="deb1a36e1b6c8c0baff85efb47da616a5394342da673679463e33a0d1c2e5958" exitCode=0 Sep 11 20:00:01.748623 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:01.748520 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" event={"ID":"d0f9a898-14fe-4d54-9ddf-597d6b0743b3","Type":"ContainerDied","Data":"deb1a36e1b6c8c0baff85efb47da616a5394342da673679463e33a0d1c2e5958"} Sep 11 20:00:02.914282 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:02.914240 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" Sep 11 20:00:02.930020 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:02.929993 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" Sep 11 20:00:02.978399 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:02.978366 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-5a0309" Sep 11 20:00:03.087706 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.087602 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdms8\" (UniqueName: \"kubernetes.io/projected/53615f32-af7a-48ce-9e18-ee4d6788730b-kube-api-access-hdms8\") pod \"53615f32-af7a-48ce-9e18-ee4d6788730b\" (UID: \"53615f32-af7a-48ce-9e18-ee4d6788730b\") " Sep 11 20:00:03.089904 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.089873 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53615f32-af7a-48ce-9e18-ee4d6788730b-kube-api-access-hdms8" (OuterVolumeSpecName: "kube-api-access-hdms8") pod "53615f32-af7a-48ce-9e18-ee4d6788730b" (UID: "53615f32-af7a-48ce-9e18-ee4d6788730b"). InnerVolumeSpecName "kube-api-access-hdms8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:00:03.188542 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.188504 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdms8\" (UniqueName: \"kubernetes.io/projected/53615f32-af7a-48ce-9e18-ee4d6788730b-kube-api-access-hdms8\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:00:03.224839 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.224800 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53615f32-af7a-48ce-9e18-ee4d6788730b" path="/var/lib/kubelet/pods/53615f32-af7a-48ce-9e18-ee4d6788730b/volumes" Sep 11 20:00:03.765138 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.765108 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" Sep 11 20:00:03.765303 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.765148 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh" event={"ID":"d0f9a898-14fe-4d54-9ddf-597d6b0743b3","Type":"ContainerDied","Data":"d76d6290035c585cd16e7713828c9516361dc8afdda01a0922c8dd059b34a24a"} Sep 11 20:00:03.765303 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.765207 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d76d6290035c585cd16e7713828c9516361dc8afdda01a0922c8dd059b34a24a" Sep 11 20:00:03.766662 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.766645 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" Sep 11 20:00:03.766662 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.766652 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc" event={"ID":"42f74184-da9b-4932-9603-b46696237ae7","Type":"ContainerDied","Data":"ba2400d788034c8c5e6e190db376f9544794167950abbdd9d04a6da5dad9c417"} Sep 11 20:00:03.766854 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.766700 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba2400d788034c8c5e6e190db376f9544794167950abbdd9d04a6da5dad9c417" Sep 11 20:00:03.768226 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.768203 2559 generic.go:356] "Generic (PLEG): container finished" podID="53615f32-af7a-48ce-9e18-ee4d6788730b" containerID="2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5" exitCode=137 Sep 11 20:00:03.768330 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.768240 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-5a0309" Sep 11 20:00:03.768330 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.768278 2559 scope.go:117] "RemoveContainer" containerID="2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5" Sep 11 20:00:03.777997 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.777978 2559 scope.go:117] "RemoveContainer" containerID="2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5" Sep 11 20:00:03.778275 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:00:03.778258 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5\": container with ID starting with 2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5 not found: ID does not exist" containerID="2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5" Sep 11 20:00:03.778326 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:03.778284 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5"} err="failed to get container status \"2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5\": rpc error: code = NotFound desc = could not find container \"2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5\": container with ID starting with 2e8d799e1f55663bb1840cc8ba4e65f0383daa24748b70f77ab50f447dbce0e5 not found: ID does not exist" Sep 11 20:00:05.883050 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:05.883009 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:00:15.482842 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:00:15.482781 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:01:02.770835 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:02.770796 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:01:07.802650 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:07.802614 2559 scope.go:117] "RemoveContainer" containerID="ec24e6da8ff4964bfb040e99cc2cf0fda7b765f65b4ccaaa283be3f8f2966c1c" Sep 11 20:01:16.458633 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:16.458598 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:01:28.383735 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.383698 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7710-9fe2c1"] Sep 11 20:01:28.384118 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.384103 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42f74184-da9b-4932-9603-b46696237ae7" containerName="cleanup" Sep 11 20:01:28.384163 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.384120 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="42f74184-da9b-4932-9603-b46696237ae7" containerName="cleanup" Sep 11 20:01:28.384163 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.384144 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0f9a898-14fe-4d54-9ddf-597d6b0743b3" containerName="cleanup" Sep 11 20:01:28.384163 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.384149 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0f9a898-14fe-4d54-9ddf-597d6b0743b3" containerName="cleanup" Sep 11 20:01:28.384264 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.384206 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="42f74184-da9b-4932-9603-b46696237ae7" containerName="cleanup" Sep 11 20:01:28.384264 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.384218 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0f9a898-14fe-4d54-9ddf-597d6b0743b3" containerName="cleanup" Sep 11 20:01:28.387124 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.387108 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-9fe2c1" Sep 11 20:01:28.390612 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.390588 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xq6qr\"" Sep 11 20:01:28.392642 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.392621 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-9fe2c1"] Sep 11 20:01:28.483037 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.482995 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdqtk\" (UniqueName: \"kubernetes.io/projected/5272c1dc-197a-4d12-ba52-4d3fbcfedd74-kube-api-access-kdqtk\") pod \"test-curl-7710-9fe2c1\" (UID: \"5272c1dc-197a-4d12-ba52-4d3fbcfedd74\") " pod="opendatahub/test-curl-7710-9fe2c1" Sep 11 20:01:28.583677 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.583637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdqtk\" (UniqueName: \"kubernetes.io/projected/5272c1dc-197a-4d12-ba52-4d3fbcfedd74-kube-api-access-kdqtk\") pod \"test-curl-7710-9fe2c1\" (UID: \"5272c1dc-197a-4d12-ba52-4d3fbcfedd74\") " pod="opendatahub/test-curl-7710-9fe2c1" Sep 11 20:01:28.593183 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.593153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdqtk\" (UniqueName: \"kubernetes.io/projected/5272c1dc-197a-4d12-ba52-4d3fbcfedd74-kube-api-access-kdqtk\") pod \"test-curl-7710-9fe2c1\" (UID: \"5272c1dc-197a-4d12-ba52-4d3fbcfedd74\") " pod="opendatahub/test-curl-7710-9fe2c1" Sep 11 20:01:28.697630 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.697545 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-9fe2c1" Sep 11 20:01:28.971832 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:28.971795 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-9fe2c1"] Sep 11 20:01:29.304398 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:29.304364 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-9fe2c1" event={"ID":"5272c1dc-197a-4d12-ba52-4d3fbcfedd74","Type":"ContainerStarted","Data":"3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658"} Sep 11 20:01:29.304583 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:29.304403 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-9fe2c1" event={"ID":"5272c1dc-197a-4d12-ba52-4d3fbcfedd74","Type":"ContainerStarted","Data":"4cd8013391ebcddc004f7c08f9ed0c3158f28c31d5419dcc123b45dc4f16349d"} Sep 11 20:01:29.322133 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:29.322070 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7710-9fe2c1" podStartSLOduration=1.322053701 podStartE2EDuration="1.322053701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:29.319789369 +0000 UTC m=+2122.592296568" watchObservedRunningTime="2026-09-11 20:01:29.322053701 +0000 UTC m=+2122.594560901" Sep 11 20:01:29.905586 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:29.905551 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7710-9fe2c1"] Sep 11 20:01:29.913697 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:29.913667 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7710-9fe2c1"] Sep 11 20:01:30.017306 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.017272 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7710-7d6239"] Sep 11 20:01:30.017721 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.017706 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5272c1dc-197a-4d12-ba52-4d3fbcfedd74" containerName="test-curl-7710-9fe2c1" Sep 11 20:01:30.017789 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.017724 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5272c1dc-197a-4d12-ba52-4d3fbcfedd74" containerName="test-curl-7710-9fe2c1" Sep 11 20:01:30.017827 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.017794 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5272c1dc-197a-4d12-ba52-4d3fbcfedd74" containerName="test-curl-7710-9fe2c1" Sep 11 20:01:30.020863 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.020848 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:30.024831 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.024805 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-7d6239"] Sep 11 20:01:30.096671 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.096634 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ml6\" (UniqueName: \"kubernetes.io/projected/63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8-kube-api-access-h8ml6\") pod \"test-curl-7710-7d6239\" (UID: \"63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8\") " pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:30.197898 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.197856 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ml6\" (UniqueName: \"kubernetes.io/projected/63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8-kube-api-access-h8ml6\") pod \"test-curl-7710-7d6239\" (UID: \"63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8\") " pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:30.207115 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.207088 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ml6\" (UniqueName: \"kubernetes.io/projected/63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8-kube-api-access-h8ml6\") pod \"test-curl-7710-7d6239\" (UID: \"63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8\") " pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:30.311572 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:01:30.311492 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7710-9fe2c1" container="test-curl-7710-9fe2c1" Sep 11 20:01:30.331243 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.331218 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:30.603540 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:30.603504 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7710-7d6239"] Sep 11 20:01:30.604077 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:01:30.604044 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63fc212d_cbd4_4e9b_8d06_e3cfd2ba12e8.slice/crio-6329ba91d8bf262e39e344d46414c1c7518a1948c39aad2773a3ea31fc9e3e57 WatchSource:0}: Error finding container 6329ba91d8bf262e39e344d46414c1c7518a1948c39aad2773a3ea31fc9e3e57: Status 404 returned error can't find the container with id 6329ba91d8bf262e39e344d46414c1c7518a1948c39aad2773a3ea31fc9e3e57 Sep 11 20:01:31.318537 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:31.318505 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-7d6239" event={"ID":"63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8","Type":"ContainerStarted","Data":"875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58"} Sep 11 20:01:31.318537 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:31.318548 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7710-7d6239" event={"ID":"63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8","Type":"ContainerStarted","Data":"6329ba91d8bf262e39e344d46414c1c7518a1948c39aad2773a3ea31fc9e3e57"} Sep 11 20:01:31.319004 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:31.318589 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7710-9fe2c1" podUID="5272c1dc-197a-4d12-ba52-4d3fbcfedd74" containerName="test-curl-7710-9fe2c1" containerID="cri-o://3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658" gracePeriod=2 Sep 11 20:01:31.335521 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:31.335476 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7710-7d6239" podStartSLOduration=1.335460944 podStartE2EDuration="1.335460944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:01:31.334519079 +0000 UTC m=+2124.607026279" watchObservedRunningTime="2026-09-11 20:01:31.335460944 +0000 UTC m=+2124.607968138" Sep 11 20:01:31.952779 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:31.952745 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7710-7d6239"] Sep 11 20:01:31.963829 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:31.963758 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7710-7d6239"] Sep 11 20:01:32.324733 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:01:32.324706 2559 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7710-7d6239" container="test-curl-7710-7d6239" Sep 11 20:01:32.474387 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:32.474352 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:01:33.330968 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:33.330933 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7710-7d6239" podUID="63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" containerName="test-curl-7710-7d6239" containerID="cri-o://875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58" gracePeriod=2 Sep 11 20:01:33.493922 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:33.493896 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-9fe2c1" Sep 11 20:01:33.628060 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:33.627971 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdqtk\" (UniqueName: \"kubernetes.io/projected/5272c1dc-197a-4d12-ba52-4d3fbcfedd74-kube-api-access-kdqtk\") pod \"5272c1dc-197a-4d12-ba52-4d3fbcfedd74\" (UID: \"5272c1dc-197a-4d12-ba52-4d3fbcfedd74\") " Sep 11 20:01:33.630234 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:33.630207 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5272c1dc-197a-4d12-ba52-4d3fbcfedd74-kube-api-access-kdqtk" (OuterVolumeSpecName: "kube-api-access-kdqtk") pod "5272c1dc-197a-4d12-ba52-4d3fbcfedd74" (UID: "5272c1dc-197a-4d12-ba52-4d3fbcfedd74"). InnerVolumeSpecName "kube-api-access-kdqtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:33.728906 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:33.728862 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdqtk\" (UniqueName: \"kubernetes.io/projected/5272c1dc-197a-4d12-ba52-4d3fbcfedd74-kube-api-access-kdqtk\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:01:34.338304 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:34.338268 2559 generic.go:356] "Generic (PLEG): container finished" podID="5272c1dc-197a-4d12-ba52-4d3fbcfedd74" containerID="3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658" exitCode=137 Sep 11 20:01:34.338750 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:34.338312 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-9fe2c1" Sep 11 20:01:34.338750 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:34.338323 2559 scope.go:117] "RemoveContainer" containerID="3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658" Sep 11 20:01:34.349549 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:01:34.349518 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7710-9fe2c1\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="5272c1dc-197a-4d12-ba52-4d3fbcfedd74" pod="opendatahub/test-curl-7710-9fe2c1" Sep 11 20:01:34.351037 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:34.351015 2559 scope.go:117] "RemoveContainer" containerID="3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658" Sep 11 20:01:34.351343 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:01:34.351322 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658\": container with ID starting with 3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658 not found: ID does not exist" containerID="3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658" Sep 11 20:01:34.351429 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:34.351358 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658"} err="failed to get container status \"3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658\": rpc error: code = NotFound desc = could not find container \"3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658\": container with ID starting with 3b74be8d273d652d43457a4952b188767058088f31f11f44c911212131935658 not found: ID does not exist" Sep 11 20:01:35.224411 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:35.224375 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5272c1dc-197a-4d12-ba52-4d3fbcfedd74" path="/var/lib/kubelet/pods/5272c1dc-197a-4d12-ba52-4d3fbcfedd74/volumes" Sep 11 20:01:35.492773 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:35.492742 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:35.646778 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:35.646741 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8ml6\" (UniqueName: \"kubernetes.io/projected/63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8-kube-api-access-h8ml6\") pod \"63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8\" (UID: \"63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8\") " Sep 11 20:01:35.649059 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:35.649024 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8-kube-api-access-h8ml6" (OuterVolumeSpecName: "kube-api-access-h8ml6") pod "63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" (UID: "63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8"). InnerVolumeSpecName "kube-api-access-h8ml6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:01:35.748219 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:35.748185 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ml6\" (UniqueName: \"kubernetes.io/projected/63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8-kube-api-access-h8ml6\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:01:36.353053 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:36.353021 2559 generic.go:356] "Generic (PLEG): container finished" podID="63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" containerID="875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58" exitCode=137 Sep 11 20:01:36.353262 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:36.353071 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:36.353262 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:36.353132 2559 scope.go:117] "RemoveContainer" containerID="875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58" Sep 11 20:01:36.362924 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:36.362899 2559 scope.go:117] "RemoveContainer" containerID="875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58" Sep 11 20:01:36.363245 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:01:36.363221 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58\": container with ID starting with 875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58 not found: ID does not exist" containerID="875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58" Sep 11 20:01:36.363314 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:36.363258 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58"} err="failed to get container status \"875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58\": rpc error: code = NotFound desc = could not find container \"875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58\": container with ID starting with 875763e0a7e42e5c91b6179fc098540dae6d10ea0934307b56af84de971c7c58 not found: ID does not exist" Sep 11 20:01:36.363985 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:01:36.363962 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7710-7d6239\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:37.223211 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:01:37.223168 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7710-7d6239\" is forbidden: User \"system:node:ip-10-0-137-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-97.ec2.internal' and this object" podUID="63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" pod="opendatahub/test-curl-7710-7d6239" Sep 11 20:01:37.224291 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:37.224271 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" path="/var/lib/kubelet/pods/63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8/volumes" Sep 11 20:01:44.363775 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:01:44.363733 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:02:08.461664 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:08.461631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:02:27.264303 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:27.264260 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:02:36.757072 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:36.757035 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:02:45.954795 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:45.954756 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:02:50.063117 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.063082 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472"] Sep 11 20:02:50.063552 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.063485 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" containerName="test-curl-7710-7d6239" Sep 11 20:02:50.063552 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.063498 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" containerName="test-curl-7710-7d6239" Sep 11 20:02:50.063643 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.063572 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="63fc212d-cbd4-4e9b-8d06-e3cfd2ba12e8" containerName="test-curl-7710-7d6239" Sep 11 20:02:50.066796 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.066776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.069603 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.069580 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-81ca1f\"" Sep 11 20:02:50.069603 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.069595 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-81ca1f-dockercfg-f68z6\"" Sep 11 20:02:50.078189 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.078162 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472"] Sep 11 20:02:50.080705 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.080679 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8"] Sep 11 20:02:50.084134 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.084117 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.095483 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.095426 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8"] Sep 11 20:02:50.136324 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.136271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr25b\" (UniqueName: \"kubernetes.io/projected/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-kube-api-access-zr25b\") pod \"payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472\" (UID: \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.136528 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.136390 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfxns\" (UniqueName: \"kubernetes.io/projected/dbd13854-28f7-4719-8b31-a903874d800f-kube-api-access-hfxns\") pod \"payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8\" (UID: \"dbd13854-28f7-4719-8b31-a903874d800f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.136528 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.136416 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472\" (UID: \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.136528 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.136501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd13854-28f7-4719-8b31-a903874d800f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8\" (UID: \"dbd13854-28f7-4719-8b31-a903874d800f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.237595 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.237547 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfxns\" (UniqueName: \"kubernetes.io/projected/dbd13854-28f7-4719-8b31-a903874d800f-kube-api-access-hfxns\") pod \"payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8\" (UID: \"dbd13854-28f7-4719-8b31-a903874d800f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.237595 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.237607 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472\" (UID: \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.237917 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.237875 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd13854-28f7-4719-8b31-a903874d800f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8\" (UID: \"dbd13854-28f7-4719-8b31-a903874d800f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.238065 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.238024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr25b\" (UniqueName: \"kubernetes.io/projected/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-kube-api-access-zr25b\") pod \"payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472\" (UID: \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.238495 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.238473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472\" (UID: \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.238675 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.238657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd13854-28f7-4719-8b31-a903874d800f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8\" (UID: \"dbd13854-28f7-4719-8b31-a903874d800f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.247052 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.247031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfxns\" (UniqueName: \"kubernetes.io/projected/dbd13854-28f7-4719-8b31-a903874d800f-kube-api-access-hfxns\") pod \"payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8\" (UID: \"dbd13854-28f7-4719-8b31-a903874d800f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.249606 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.249582 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr25b\" (UniqueName: \"kubernetes.io/projected/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-kube-api-access-zr25b\") pod \"payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472\" (UID: \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.378991 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.378907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.395797 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.395769 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.656107 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.656074 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472"] Sep 11 20:02:50.657508 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:02:50.657477 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc6e41e_117f_4521_a04b_3ccbb64a2d26.slice/crio-3d720fd94aeeadccf6e5fed96b61030c0663a57b066da9edae2a13ed2405f7e3 WatchSource:0}: Error finding container 3d720fd94aeeadccf6e5fed96b61030c0663a57b066da9edae2a13ed2405f7e3: Status 404 returned error can't find the container with id 3d720fd94aeeadccf6e5fed96b61030c0663a57b066da9edae2a13ed2405f7e3 Sep 11 20:02:50.682510 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.682487 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8"] Sep 11 20:02:50.698818 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:02:50.698789 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd13854_28f7_4719_8b31_a903874d800f.slice/crio-8bf5adc422b96b45df207b85990a3a84e20fcab1b44c0e8a29362438f561e791 WatchSource:0}: Error finding container 8bf5adc422b96b45df207b85990a3a84e20fcab1b44c0e8a29362438f561e791: Status 404 returned error can't find the container with id 8bf5adc422b96b45df207b85990a3a84e20fcab1b44c0e8a29362438f561e791 Sep 11 20:02:50.815655 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.815472 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" event={"ID":"bbc6e41e-117f-4521-a04b-3ccbb64a2d26","Type":"ContainerStarted","Data":"6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128"} Sep 11 20:02:50.815655 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.815587 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:02:50.815655 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.815613 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" event={"ID":"bbc6e41e-117f-4521-a04b-3ccbb64a2d26","Type":"ContainerStarted","Data":"3d720fd94aeeadccf6e5fed96b61030c0663a57b066da9edae2a13ed2405f7e3"} Sep 11 20:02:50.817783 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.817729 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" event={"ID":"dbd13854-28f7-4719-8b31-a903874d800f","Type":"ContainerStarted","Data":"991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2"} Sep 11 20:02:50.817917 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.817794 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" event={"ID":"dbd13854-28f7-4719-8b31-a903874d800f","Type":"ContainerStarted","Data":"8bf5adc422b96b45df207b85990a3a84e20fcab1b44c0e8a29362438f561e791"} Sep 11 20:02:50.817917 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.817846 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:02:50.835125 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.835068 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" podStartSLOduration=0.835051868 podStartE2EDuration="835.051868ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:02:50.832602312 +0000 UTC m=+2204.105109513" watchObservedRunningTime="2026-09-11 20:02:50.835051868 +0000 UTC m=+2204.107559065" Sep 11 20:02:50.851166 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:02:50.851115 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" podStartSLOduration=0.851099778 podStartE2EDuration="851.099778ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:02:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:02:50.848243906 +0000 UTC m=+2204.120751106" watchObservedRunningTime="2026-09-11 20:02:50.851099778 +0000 UTC m=+2204.123606995" Sep 11 20:03:00.250133 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:00.250090 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:03:01.827272 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:01.827241 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:03:01.827731 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:01.827582 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:03:19.769140 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:19.769104 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5"] Sep 11 20:03:19.772654 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:19.772637 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:19.777373 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:19.776353 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5df0d4-dockercfg-wc4j2\"" Sep 11 20:03:19.780467 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:19.777854 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5df0d4\"" Sep 11 20:03:19.785665 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:19.785631 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5"] Sep 11 20:03:19.916025 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:19.915986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6mv\" (UniqueName: \"kubernetes.io/projected/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-kube-api-access-9g6mv\") pod \"payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5\" (UID: \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:19.916025 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:19.916025 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5\" (UID: \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:20.017093 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:20.017055 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6mv\" (UniqueName: \"kubernetes.io/projected/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-kube-api-access-9g6mv\") pod \"payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5\" (UID: \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:20.017093 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:20.017094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5\" (UID: \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:20.017850 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:20.017828 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5\" (UID: \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:20.025995 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:20.025939 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6mv\" (UniqueName: \"kubernetes.io/projected/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-kube-api-access-9g6mv\") pod \"payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5\" (UID: \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:20.087071 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:20.087038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:20.370921 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:20.370889 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5"] Sep 11 20:03:20.372030 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:03:20.372007 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc5ed0aa_c5cf_4a6e_807b_4df2fe2799d3.slice/crio-21e0bd34369d7f299b8df344919015054f5349885cdbfef1e6d1a4c512e09629 WatchSource:0}: Error finding container 21e0bd34369d7f299b8df344919015054f5349885cdbfef1e6d1a4c512e09629: Status 404 returned error can't find the container with id 21e0bd34369d7f299b8df344919015054f5349885cdbfef1e6d1a4c512e09629 Sep 11 20:03:21.018775 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:21.018720 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" event={"ID":"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3","Type":"ContainerStarted","Data":"e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d"} Sep 11 20:03:21.019213 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:21.018789 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" event={"ID":"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3","Type":"ContainerStarted","Data":"21e0bd34369d7f299b8df344919015054f5349885cdbfef1e6d1a4c512e09629"} Sep 11 20:03:21.019213 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:21.018937 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:03:21.039159 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:21.039107 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" podStartSLOduration=2.039093774 podStartE2EDuration="2.039093774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:03:21.036051123 +0000 UTC m=+2234.308558324" watchObservedRunningTime="2026-09-11 20:03:21.039093774 +0000 UTC m=+2234.311600974" Sep 11 20:03:24.859000 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:24.858957 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:03:32.027244 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:03:32.027169 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:04:25.969591 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:04:25.969554 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:04:46.952047 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:04:46.952003 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:04:53.117843 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:04:53.117801 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8_dbd13854-28f7-4719-8b31-a903874d800f/payload-processing/0.log" Sep 11 20:05:01.176643 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.176562 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n"] Sep 11 20:05:01.180477 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.180460 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:01.183172 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.183150 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-3d4be6-dockercfg-5w47h\"" Sep 11 20:05:01.183172 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.183152 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-3d4be6\"" Sep 11 20:05:01.190798 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.190779 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n"] Sep 11 20:05:01.274462 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.274411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fe51c3-b470-46f8-acc7-a048c44148b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n\" (UID: \"c6fe51c3-b470-46f8-acc7-a048c44148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:01.274628 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.274548 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wkql\" (UniqueName: \"kubernetes.io/projected/c6fe51c3-b470-46f8-acc7-a048c44148b7-kube-api-access-2wkql\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n\" (UID: \"c6fe51c3-b470-46f8-acc7-a048c44148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:01.308264 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.308230 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2"] Sep 11 20:05:01.312604 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.312586 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:01.322741 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.322718 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2"] Sep 11 20:05:01.376012 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.375971 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fe51c3-b470-46f8-acc7-a048c44148b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n\" (UID: \"c6fe51c3-b470-46f8-acc7-a048c44148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:01.376208 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.376051 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36605195-9931-40f0-82d1-e8d57b145f69-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2\" (UID: \"36605195-9931-40f0-82d1-e8d57b145f69\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:01.376208 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.376080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wkql\" (UniqueName: \"kubernetes.io/projected/c6fe51c3-b470-46f8-acc7-a048c44148b7-kube-api-access-2wkql\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n\" (UID: \"c6fe51c3-b470-46f8-acc7-a048c44148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:01.376208 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.376119 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-565g5\" (UniqueName: \"kubernetes.io/projected/36605195-9931-40f0-82d1-e8d57b145f69-kube-api-access-565g5\") pod \"payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2\" (UID: \"36605195-9931-40f0-82d1-e8d57b145f69\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:01.376713 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.376692 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fe51c3-b470-46f8-acc7-a048c44148b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n\" (UID: \"c6fe51c3-b470-46f8-acc7-a048c44148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:01.384431 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.384405 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wkql\" (UniqueName: \"kubernetes.io/projected/c6fe51c3-b470-46f8-acc7-a048c44148b7-kube-api-access-2wkql\") pod \"payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n\" (UID: \"c6fe51c3-b470-46f8-acc7-a048c44148b7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:01.477062 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.476973 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-565g5\" (UniqueName: \"kubernetes.io/projected/36605195-9931-40f0-82d1-e8d57b145f69-kube-api-access-565g5\") pod \"payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2\" (UID: \"36605195-9931-40f0-82d1-e8d57b145f69\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:01.477062 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.477058 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36605195-9931-40f0-82d1-e8d57b145f69-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2\" (UID: \"36605195-9931-40f0-82d1-e8d57b145f69\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:01.477740 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.477722 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36605195-9931-40f0-82d1-e8d57b145f69-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2\" (UID: \"36605195-9931-40f0-82d1-e8d57b145f69\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:01.486642 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.486611 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-565g5\" (UniqueName: \"kubernetes.io/projected/36605195-9931-40f0-82d1-e8d57b145f69-kube-api-access-565g5\") pod \"payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2\" (UID: \"36605195-9931-40f0-82d1-e8d57b145f69\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:01.490792 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.490775 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:01.623487 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.623432 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:01.768507 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.768475 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n"] Sep 11 20:05:01.770029 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:05:01.769997 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fe51c3_b470_46f8_acc7_a048c44148b7.slice/crio-c89a90f32dab3a96ea289288a27b2ee845753aa6c41fa73bba6df475ad86fe3d WatchSource:0}: Error finding container c89a90f32dab3a96ea289288a27b2ee845753aa6c41fa73bba6df475ad86fe3d: Status 404 returned error can't find the container with id c89a90f32dab3a96ea289288a27b2ee845753aa6c41fa73bba6df475ad86fe3d Sep 11 20:05:01.902389 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:01.902345 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2"] Sep 11 20:05:01.903286 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:05:01.903241 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36605195_9931_40f0_82d1_e8d57b145f69.slice/crio-005e82fbc0c079a5b7efa22b719d68ea36a29fb76cd2aeb614f07c9cdfbb03ee WatchSource:0}: Error finding container 005e82fbc0c079a5b7efa22b719d68ea36a29fb76cd2aeb614f07c9cdfbb03ee: Status 404 returned error can't find the container with id 005e82fbc0c079a5b7efa22b719d68ea36a29fb76cd2aeb614f07c9cdfbb03ee Sep 11 20:05:02.656068 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:02.656022 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" event={"ID":"c6fe51c3-b470-46f8-acc7-a048c44148b7","Type":"ContainerStarted","Data":"3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a"} Sep 11 20:05:02.656577 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:02.656261 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:02.656577 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:02.656319 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" event={"ID":"c6fe51c3-b470-46f8-acc7-a048c44148b7","Type":"ContainerStarted","Data":"c89a90f32dab3a96ea289288a27b2ee845753aa6c41fa73bba6df475ad86fe3d"} Sep 11 20:05:02.657855 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:02.657827 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" event={"ID":"36605195-9931-40f0-82d1-e8d57b145f69","Type":"ContainerStarted","Data":"839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028"} Sep 11 20:05:02.657982 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:02.657873 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" event={"ID":"36605195-9931-40f0-82d1-e8d57b145f69","Type":"ContainerStarted","Data":"005e82fbc0c079a5b7efa22b719d68ea36a29fb76cd2aeb614f07c9cdfbb03ee"} Sep 11 20:05:02.657982 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:02.657894 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:02.677122 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:02.677073 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" podStartSLOduration=1.677058484 podStartE2EDuration="1.677058484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:02.673752638 +0000 UTC m=+2335.946259841" watchObservedRunningTime="2026-09-11 20:05:02.677058484 +0000 UTC m=+2335.949565729" Sep 11 20:05:02.690720 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:02.690669 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" podStartSLOduration=1.690650867 podStartE2EDuration="1.690650867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:02.690609155 +0000 UTC m=+2335.963116358" watchObservedRunningTime="2026-09-11 20:05:02.690650867 +0000 UTC m=+2335.963158069" Sep 11 20:05:07.360223 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:07.360184 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:05:07.866548 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:07.866515 2559 scope.go:117] "RemoveContainer" containerID="e3f9cd46ed7d8170f6e7af6dc4b5381bd217c352b7a0544152ae27a0c9cffd5c" Sep 11 20:05:07.876280 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:07.876256 2559 scope.go:117] "RemoveContainer" containerID="431e5bbe1a3e0af83b5b31135d1250d48d329f5b39473be29d8800cb3939a01b" Sep 11 20:05:07.885337 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:07.885316 2559 scope.go:117] "RemoveContainer" containerID="2de4af8624ee862117c72b6c1dfdcb986ec954206855e7009f496a2235170e01" Sep 11 20:05:07.893677 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:07.893657 2559 scope.go:117] "RemoveContainer" containerID="5f2ba40b47363d4cfc4845719a1d17dbebef7d8d61696adaa919535b680f2235" Sep 11 20:05:13.665756 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:13.665713 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:13.666140 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:13.666018 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:27.864808 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:27.864763 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz"] Sep 11 20:05:27.868521 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:27.868497 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" Sep 11 20:05:27.871252 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:27.871225 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 20:05:27.871383 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:27.871321 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 20:05:27.876504 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:27.876481 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz"] Sep 11 20:05:28.027673 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:28.027638 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb48dae-d39f-4d21-9443-5c0a8589e4aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz\" (UID: \"6eb48dae-d39f-4d21-9443-5c0a8589e4aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" Sep 11 20:05:28.129140 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:28.129036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb48dae-d39f-4d21-9443-5c0a8589e4aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz\" (UID: \"6eb48dae-d39f-4d21-9443-5c0a8589e4aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" Sep 11 20:05:28.129728 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:28.129703 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb48dae-d39f-4d21-9443-5c0a8589e4aa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz\" (UID: \"6eb48dae-d39f-4d21-9443-5c0a8589e4aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" Sep 11 20:05:28.179638 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:28.179607 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" Sep 11 20:05:28.453913 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:28.453886 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz"] Sep 11 20:05:28.455581 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:05:28.455552 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eb48dae_d39f_4d21_9443_5c0a8589e4aa.slice/crio-f5d1f8de6a406e36e6886a2bd064adc6c1b0051347b6d4a2288d4217418cd96e WatchSource:0}: Error finding container f5d1f8de6a406e36e6886a2bd064adc6c1b0051347b6d4a2288d4217418cd96e: Status 404 returned error can't find the container with id f5d1f8de6a406e36e6886a2bd064adc6c1b0051347b6d4a2288d4217418cd96e Sep 11 20:05:28.826800 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:28.826758 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" event={"ID":"6eb48dae-d39f-4d21-9443-5c0a8589e4aa","Type":"ContainerStarted","Data":"6e2c57f7df725113eed4d6a67439a1a39e91232d4095db371613d6ac64d2f572"} Sep 11 20:05:28.826972 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:28.826822 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" event={"ID":"6eb48dae-d39f-4d21-9443-5c0a8589e4aa","Type":"ContainerStarted","Data":"f5d1f8de6a406e36e6886a2bd064adc6c1b0051347b6d4a2288d4217418cd96e"} Sep 11 20:05:28.845418 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:28.845368 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" podStartSLOduration=1.845351783 podStartE2EDuration="1.845351783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:05:28.842961427 +0000 UTC m=+2362.115468629" watchObservedRunningTime="2026-09-11 20:05:28.845351783 +0000 UTC m=+2362.117858983" Sep 11 20:05:29.836220 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:29.836192 2559 generic.go:356] "Generic (PLEG): container finished" podID="6eb48dae-d39f-4d21-9443-5c0a8589e4aa" containerID="6e2c57f7df725113eed4d6a67439a1a39e91232d4095db371613d6ac64d2f572" exitCode=0 Sep 11 20:05:29.836609 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:29.836278 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" event={"ID":"6eb48dae-d39f-4d21-9443-5c0a8589e4aa","Type":"ContainerDied","Data":"6e2c57f7df725113eed4d6a67439a1a39e91232d4095db371613d6ac64d2f572"} Sep 11 20:05:30.978156 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:30.978127 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" Sep 11 20:05:31.060832 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:31.060798 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb48dae-d39f-4d21-9443-5c0a8589e4aa-maas-api-service-ca\") pod \"6eb48dae-d39f-4d21-9443-5c0a8589e4aa\" (UID: \"6eb48dae-d39f-4d21-9443-5c0a8589e4aa\") " Sep 11 20:05:31.061208 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:31.061185 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eb48dae-d39f-4d21-9443-5c0a8589e4aa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6eb48dae-d39f-4d21-9443-5c0a8589e4aa" (UID: "6eb48dae-d39f-4d21-9443-5c0a8589e4aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:05:31.161944 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:31.161868 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eb48dae-d39f-4d21-9443-5c0a8589e4aa-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:05:31.850253 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:31.850221 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" Sep 11 20:05:31.850433 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:31.850262 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz" event={"ID":"6eb48dae-d39f-4d21-9443-5c0a8589e4aa","Type":"ContainerDied","Data":"f5d1f8de6a406e36e6886a2bd064adc6c1b0051347b6d4a2288d4217418cd96e"} Sep 11 20:05:31.850433 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:31.850312 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5d1f8de6a406e36e6886a2bd064adc6c1b0051347b6d4a2288d4217418cd96e" Sep 11 20:05:38.947392 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:38.947350 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2"] Sep 11 20:05:38.947993 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:38.947678 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" podUID="36605195-9931-40f0-82d1-e8d57b145f69" containerName="payload-processing" containerID="cri-o://839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028" gracePeriod=30 Sep 11 20:05:38.958367 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:38.958336 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n"] Sep 11 20:05:38.958637 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:38.958605 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" podUID="c6fe51c3-b470-46f8-acc7-a048c44148b7" containerName="payload-pre-processing" containerID="cri-o://3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a" gracePeriod=30 Sep 11 20:05:39.151854 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.151822 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz"] Sep 11 20:05:39.154249 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.154227 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-3d4be6-s9wgz"] Sep 11 20:05:39.224674 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.224644 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eb48dae-d39f-4d21-9443-5c0a8589e4aa" path="/var/lib/kubelet/pods/6eb48dae-d39f-4d21-9443-5c0a8589e4aa/volumes" Sep 11 20:05:39.238464 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.238423 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:39.241216 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.241199 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:39.336304 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.336269 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36605195-9931-40f0-82d1-e8d57b145f69-plugins-config-volume\") pod \"36605195-9931-40f0-82d1-e8d57b145f69\" (UID: \"36605195-9931-40f0-82d1-e8d57b145f69\") " Sep 11 20:05:39.336530 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.336330 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-565g5\" (UniqueName: \"kubernetes.io/projected/36605195-9931-40f0-82d1-e8d57b145f69-kube-api-access-565g5\") pod \"36605195-9931-40f0-82d1-e8d57b145f69\" (UID: \"36605195-9931-40f0-82d1-e8d57b145f69\") " Sep 11 20:05:39.336676 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.336646 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36605195-9931-40f0-82d1-e8d57b145f69-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "36605195-9931-40f0-82d1-e8d57b145f69" (UID: "36605195-9931-40f0-82d1-e8d57b145f69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:05:39.338700 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.338676 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36605195-9931-40f0-82d1-e8d57b145f69-kube-api-access-565g5" (OuterVolumeSpecName: "kube-api-access-565g5") pod "36605195-9931-40f0-82d1-e8d57b145f69" (UID: "36605195-9931-40f0-82d1-e8d57b145f69"). InnerVolumeSpecName "kube-api-access-565g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:05:39.437166 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.437130 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fe51c3-b470-46f8-acc7-a048c44148b7-plugins-config-volume\") pod \"c6fe51c3-b470-46f8-acc7-a048c44148b7\" (UID: \"c6fe51c3-b470-46f8-acc7-a048c44148b7\") " Sep 11 20:05:39.437358 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.437197 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wkql\" (UniqueName: \"kubernetes.io/projected/c6fe51c3-b470-46f8-acc7-a048c44148b7-kube-api-access-2wkql\") pod \"c6fe51c3-b470-46f8-acc7-a048c44148b7\" (UID: \"c6fe51c3-b470-46f8-acc7-a048c44148b7\") " Sep 11 20:05:39.437426 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.437371 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-565g5\" (UniqueName: \"kubernetes.io/projected/36605195-9931-40f0-82d1-e8d57b145f69-kube-api-access-565g5\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:05:39.437426 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.437381 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36605195-9931-40f0-82d1-e8d57b145f69-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:05:39.437633 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.437608 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fe51c3-b470-46f8-acc7-a048c44148b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c6fe51c3-b470-46f8-acc7-a048c44148b7" (UID: "c6fe51c3-b470-46f8-acc7-a048c44148b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:05:39.439404 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.439382 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6fe51c3-b470-46f8-acc7-a048c44148b7-kube-api-access-2wkql" (OuterVolumeSpecName: "kube-api-access-2wkql") pod "c6fe51c3-b470-46f8-acc7-a048c44148b7" (UID: "c6fe51c3-b470-46f8-acc7-a048c44148b7"). InnerVolumeSpecName "kube-api-access-2wkql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:05:39.538457 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.538410 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6fe51c3-b470-46f8-acc7-a048c44148b7-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:05:39.538457 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.538457 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wkql\" (UniqueName: \"kubernetes.io/projected/c6fe51c3-b470-46f8-acc7-a048c44148b7-kube-api-access-2wkql\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:05:39.905661 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.905576 2559 generic.go:356] "Generic (PLEG): container finished" podID="c6fe51c3-b470-46f8-acc7-a048c44148b7" containerID="3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a" exitCode=0 Sep 11 20:05:39.905661 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.905625 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" Sep 11 20:05:39.905897 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.905659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" event={"ID":"c6fe51c3-b470-46f8-acc7-a048c44148b7","Type":"ContainerDied","Data":"3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a"} Sep 11 20:05:39.905897 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.905714 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n" event={"ID":"c6fe51c3-b470-46f8-acc7-a048c44148b7","Type":"ContainerDied","Data":"c89a90f32dab3a96ea289288a27b2ee845753aa6c41fa73bba6df475ad86fe3d"} Sep 11 20:05:39.905897 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.905738 2559 scope.go:117] "RemoveContainer" containerID="3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a" Sep 11 20:05:39.907407 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.907385 2559 generic.go:356] "Generic (PLEG): container finished" podID="36605195-9931-40f0-82d1-e8d57b145f69" containerID="839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028" exitCode=0 Sep 11 20:05:39.907561 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.907431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" event={"ID":"36605195-9931-40f0-82d1-e8d57b145f69","Type":"ContainerDied","Data":"839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028"} Sep 11 20:05:39.907561 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.907491 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" event={"ID":"36605195-9931-40f0-82d1-e8d57b145f69","Type":"ContainerDied","Data":"005e82fbc0c079a5b7efa22b719d68ea36a29fb76cd2aeb614f07c9cdfbb03ee"} Sep 11 20:05:39.907561 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.907503 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2" Sep 11 20:05:39.917396 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.917266 2559 scope.go:117] "RemoveContainer" containerID="3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a" Sep 11 20:05:39.917704 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:05:39.917685 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a\": container with ID starting with 3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a not found: ID does not exist" containerID="3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a" Sep 11 20:05:39.917780 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.917712 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a"} err="failed to get container status \"3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a\": rpc error: code = NotFound desc = could not find container \"3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a\": container with ID starting with 3c457bd885ff7aaccfbe698c28b7c347d513a860aeee754086dbf0e04c472b7a not found: ID does not exist" Sep 11 20:05:39.917780 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.917730 2559 scope.go:117] "RemoveContainer" containerID="839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028" Sep 11 20:05:39.926595 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.926577 2559 scope.go:117] "RemoveContainer" containerID="839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028" Sep 11 20:05:39.926889 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:05:39.926871 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028\": container with ID starting with 839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028 not found: ID does not exist" containerID="839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028" Sep 11 20:05:39.926933 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.926900 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028"} err="failed to get container status \"839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028\": rpc error: code = NotFound desc = could not find container \"839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028\": container with ID starting with 839bfa83dba60a7cef86f2ed16553d93b5a4da00781fbdc341e217ef1bed4028 not found: ID does not exist" Sep 11 20:05:39.931504 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.931482 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n"] Sep 11 20:05:39.935828 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.935805 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-3d4be6-6bb47974db-7mx9n"] Sep 11 20:05:39.946675 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.946651 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2"] Sep 11 20:05:39.949917 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:39.949894 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-3d4be6-784848bb6c-9sgx2"] Sep 11 20:05:41.224543 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:41.224505 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36605195-9931-40f0-82d1-e8d57b145f69" path="/var/lib/kubelet/pods/36605195-9931-40f0-82d1-e8d57b145f69/volumes" Sep 11 20:05:41.224911 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:41.224854 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6fe51c3-b470-46f8-acc7-a048c44148b7" path="/var/lib/kubelet/pods/c6fe51c3-b470-46f8-acc7-a048c44148b7/volumes" Sep 11 20:05:43.253320 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:43.253286 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:05:49.155914 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.155880 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb"] Sep 11 20:05:49.156313 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156300 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eb48dae-d39f-4d21-9443-5c0a8589e4aa" containerName="revoke-keys" Sep 11 20:05:49.156313 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156315 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eb48dae-d39f-4d21-9443-5c0a8589e4aa" containerName="revoke-keys" Sep 11 20:05:49.156392 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156326 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36605195-9931-40f0-82d1-e8d57b145f69" containerName="payload-processing" Sep 11 20:05:49.156392 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156332 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="36605195-9931-40f0-82d1-e8d57b145f69" containerName="payload-processing" Sep 11 20:05:49.156392 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156351 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6fe51c3-b470-46f8-acc7-a048c44148b7" containerName="payload-pre-processing" Sep 11 20:05:49.156392 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156356 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6fe51c3-b470-46f8-acc7-a048c44148b7" containerName="payload-pre-processing" Sep 11 20:05:49.156532 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156417 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="36605195-9931-40f0-82d1-e8d57b145f69" containerName="payload-processing" Sep 11 20:05:49.156532 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156425 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eb48dae-d39f-4d21-9443-5c0a8589e4aa" containerName="revoke-keys" Sep 11 20:05:49.156532 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.156449 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6fe51c3-b470-46f8-acc7-a048c44148b7" containerName="payload-pre-processing" Sep 11 20:05:49.159589 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.159568 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" Sep 11 20:05:49.162039 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.162017 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 20:05:49.162140 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.162123 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 20:05:49.175241 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.175209 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb"] Sep 11 20:05:49.226012 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.225980 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f06fcb-d73f-4f53-b352-1d7f7a66566d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb\" (UID: \"65f06fcb-d73f-4f53-b352-1d7f7a66566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" Sep 11 20:05:49.327096 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.327056 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f06fcb-d73f-4f53-b352-1d7f7a66566d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb\" (UID: \"65f06fcb-d73f-4f53-b352-1d7f7a66566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" Sep 11 20:05:49.327878 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.327853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f06fcb-d73f-4f53-b352-1d7f7a66566d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb\" (UID: \"65f06fcb-d73f-4f53-b352-1d7f7a66566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" Sep 11 20:05:49.479591 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.479502 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" Sep 11 20:05:49.751643 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.751620 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb"] Sep 11 20:05:49.754329 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:05:49.754294 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f06fcb_d73f_4f53_b352_1d7f7a66566d.slice/crio-18e3a7bab3b84c6a2950ff7db4ce00926ae6bf036479265f078941fe4cb3668b WatchSource:0}: Error finding container 18e3a7bab3b84c6a2950ff7db4ce00926ae6bf036479265f078941fe4cb3668b: Status 404 returned error can't find the container with id 18e3a7bab3b84c6a2950ff7db4ce00926ae6bf036479265f078941fe4cb3668b Sep 11 20:05:49.981163 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:49.981133 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" event={"ID":"65f06fcb-d73f-4f53-b352-1d7f7a66566d","Type":"ContainerStarted","Data":"18e3a7bab3b84c6a2950ff7db4ce00926ae6bf036479265f078941fe4cb3668b"} Sep 11 20:05:50.989877 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:50.989844 2559 generic.go:356] "Generic (PLEG): container finished" podID="65f06fcb-d73f-4f53-b352-1d7f7a66566d" containerID="773dc9add8afe8e3557672fbe611ee661c52bebce5c4b903b578d640dcca7e2a" exitCode=0 Sep 11 20:05:50.990257 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:50.989923 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" event={"ID":"65f06fcb-d73f-4f53-b352-1d7f7a66566d","Type":"ContainerDied","Data":"773dc9add8afe8e3557672fbe611ee661c52bebce5c4b903b578d640dcca7e2a"} Sep 11 20:05:52.133610 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:52.133585 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" Sep 11 20:05:52.253512 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:52.253396 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f06fcb-d73f-4f53-b352-1d7f7a66566d-maas-api-service-ca\") pod \"65f06fcb-d73f-4f53-b352-1d7f7a66566d\" (UID: \"65f06fcb-d73f-4f53-b352-1d7f7a66566d\") " Sep 11 20:05:52.253758 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:52.253732 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f06fcb-d73f-4f53-b352-1d7f7a66566d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65f06fcb-d73f-4f53-b352-1d7f7a66566d" (UID: "65f06fcb-d73f-4f53-b352-1d7f7a66566d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:05:52.354608 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:52.354568 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65f06fcb-d73f-4f53-b352-1d7f7a66566d-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:05:53.004298 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:53.004262 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" Sep 11 20:05:53.004510 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:53.004311 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb" event={"ID":"65f06fcb-d73f-4f53-b352-1d7f7a66566d","Type":"ContainerDied","Data":"18e3a7bab3b84c6a2950ff7db4ce00926ae6bf036479265f078941fe4cb3668b"} Sep 11 20:05:53.004510 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:05:53.004361 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18e3a7bab3b84c6a2950ff7db4ce00926ae6bf036479265f078941fe4cb3668b" Sep 11 20:06:02.558734 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:02.558691 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:06:04.975647 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:04.975613 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8"] Sep 11 20:06:04.976140 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:04.975851 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" podUID="dbd13854-28f7-4719-8b31-a903874d800f" containerName="payload-processing" containerID="cri-o://991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2" gracePeriod=30 Sep 11 20:06:05.000869 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.000799 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472"] Sep 11 20:06:05.001119 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.001043 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" podUID="bbc6e41e-117f-4521-a04b-3ccbb64a2d26" containerName="payload-pre-processing" containerID="cri-o://6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128" gracePeriod=30 Sep 11 20:06:05.170586 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.170526 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb"] Sep 11 20:06:05.173396 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.173370 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-81ca1f-w52nb"] Sep 11 20:06:05.226403 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.226317 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f06fcb-d73f-4f53-b352-1d7f7a66566d" path="/var/lib/kubelet/pods/65f06fcb-d73f-4f53-b352-1d7f7a66566d/volumes" Sep 11 20:06:05.283103 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.283080 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:06:05.285512 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.285495 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:06:05.383660 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.383630 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd13854-28f7-4719-8b31-a903874d800f-plugins-config-volume\") pod \"dbd13854-28f7-4719-8b31-a903874d800f\" (UID: \"dbd13854-28f7-4719-8b31-a903874d800f\") " Sep 11 20:06:05.383828 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.383696 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfxns\" (UniqueName: \"kubernetes.io/projected/dbd13854-28f7-4719-8b31-a903874d800f-kube-api-access-hfxns\") pod \"dbd13854-28f7-4719-8b31-a903874d800f\" (UID: \"dbd13854-28f7-4719-8b31-a903874d800f\") " Sep 11 20:06:05.383828 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.383731 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr25b\" (UniqueName: \"kubernetes.io/projected/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-kube-api-access-zr25b\") pod \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\" (UID: \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\") " Sep 11 20:06:05.383828 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.383771 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-plugins-config-volume\") pod \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\" (UID: \"bbc6e41e-117f-4521-a04b-3ccbb64a2d26\") " Sep 11 20:06:05.384135 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.384102 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd13854-28f7-4719-8b31-a903874d800f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbd13854-28f7-4719-8b31-a903874d800f" (UID: "dbd13854-28f7-4719-8b31-a903874d800f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:06:05.384244 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.384219 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbc6e41e-117f-4521-a04b-3ccbb64a2d26" (UID: "bbc6e41e-117f-4521-a04b-3ccbb64a2d26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:06:05.386622 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.386595 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd13854-28f7-4719-8b31-a903874d800f-kube-api-access-hfxns" (OuterVolumeSpecName: "kube-api-access-hfxns") pod "dbd13854-28f7-4719-8b31-a903874d800f" (UID: "dbd13854-28f7-4719-8b31-a903874d800f"). InnerVolumeSpecName "kube-api-access-hfxns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:06:05.386721 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.386629 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-kube-api-access-zr25b" (OuterVolumeSpecName: "kube-api-access-zr25b") pod "bbc6e41e-117f-4521-a04b-3ccbb64a2d26" (UID: "bbc6e41e-117f-4521-a04b-3ccbb64a2d26"). InnerVolumeSpecName "kube-api-access-zr25b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:06:05.485102 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.485010 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbd13854-28f7-4719-8b31-a903874d800f-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:06:05.485102 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.485043 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfxns\" (UniqueName: \"kubernetes.io/projected/dbd13854-28f7-4719-8b31-a903874d800f-kube-api-access-hfxns\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:06:05.485102 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.485054 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr25b\" (UniqueName: \"kubernetes.io/projected/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-kube-api-access-zr25b\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:06:05.485102 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:05.485062 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbc6e41e-117f-4521-a04b-3ccbb64a2d26-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:06:06.094512 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.094477 2559 generic.go:356] "Generic (PLEG): container finished" podID="dbd13854-28f7-4719-8b31-a903874d800f" containerID="991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2" exitCode=0 Sep 11 20:06:06.094928 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.094524 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" Sep 11 20:06:06.094928 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.094556 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" event={"ID":"dbd13854-28f7-4719-8b31-a903874d800f","Type":"ContainerDied","Data":"991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2"} Sep 11 20:06:06.094928 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.094628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8" event={"ID":"dbd13854-28f7-4719-8b31-a903874d800f","Type":"ContainerDied","Data":"8bf5adc422b96b45df207b85990a3a84e20fcab1b44c0e8a29362438f561e791"} Sep 11 20:06:06.094928 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.094662 2559 scope.go:117] "RemoveContainer" containerID="991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2" Sep 11 20:06:06.096111 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.096087 2559 generic.go:356] "Generic (PLEG): container finished" podID="bbc6e41e-117f-4521-a04b-3ccbb64a2d26" containerID="6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128" exitCode=0 Sep 11 20:06:06.096232 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.096130 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" Sep 11 20:06:06.096232 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.096166 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" event={"ID":"bbc6e41e-117f-4521-a04b-3ccbb64a2d26","Type":"ContainerDied","Data":"6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128"} Sep 11 20:06:06.096232 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.096221 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472" event={"ID":"bbc6e41e-117f-4521-a04b-3ccbb64a2d26","Type":"ContainerDied","Data":"3d720fd94aeeadccf6e5fed96b61030c0663a57b066da9edae2a13ed2405f7e3"} Sep 11 20:06:06.114326 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.114304 2559 scope.go:117] "RemoveContainer" containerID="991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2" Sep 11 20:06:06.115300 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:06:06.115276 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2\": container with ID starting with 991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2 not found: ID does not exist" containerID="991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2" Sep 11 20:06:06.115388 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.115306 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2"} err="failed to get container status \"991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2\": rpc error: code = NotFound desc = could not find container \"991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2\": container with ID starting with 991c3093313250f9f049293317d1b8c5158a15252a235d4739ef1cab650b48e2 not found: ID does not exist" Sep 11 20:06:06.115388 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.115324 2559 scope.go:117] "RemoveContainer" containerID="6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128" Sep 11 20:06:06.125530 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.125501 2559 scope.go:117] "RemoveContainer" containerID="6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128" Sep 11 20:06:06.125868 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:06:06.125849 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128\": container with ID starting with 6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128 not found: ID does not exist" containerID="6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128" Sep 11 20:06:06.125938 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.125877 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128"} err="failed to get container status \"6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128\": rpc error: code = NotFound desc = could not find container \"6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128\": container with ID starting with 6e4a4ea78187654a89369c8bacb179a43626dfeb6850fb2e8a55be13f9494128 not found: ID does not exist" Sep 11 20:06:06.128451 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.128418 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8"] Sep 11 20:06:06.133502 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.133480 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-81ca1f-6ffc957b4f-kcck8"] Sep 11 20:06:06.149325 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.149300 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472"] Sep 11 20:06:06.151503 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:06.151485 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-81ca1f-7df5dc98f8-zs472"] Sep 11 20:06:07.223860 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:07.223827 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc6e41e-117f-4521-a04b-3ccbb64a2d26" path="/var/lib/kubelet/pods/bbc6e41e-117f-4521-a04b-3ccbb64a2d26/volumes" Sep 11 20:06:07.224196 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:07.224168 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbd13854-28f7-4719-8b31-a903874d800f" path="/var/lib/kubelet/pods/dbd13854-28f7-4719-8b31-a903874d800f/volumes" Sep 11 20:06:07.931704 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:07.931670 2559 scope.go:117] "RemoveContainer" containerID="d40c8251ef290375b39e3005150b748788c595f8aea97238b005db97fe8d0ec3" Sep 11 20:06:07.941252 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:07.941219 2559 scope.go:117] "RemoveContainer" containerID="5d3575ee648900b09c88ff99a34412613bdfbad914d552b8d28193041dc87800" Sep 11 20:06:15.667606 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:15.667568 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:06:54.214505 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.214464 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc"] Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.214882 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbd13854-28f7-4719-8b31-a903874d800f" containerName="payload-processing" Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.214893 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbd13854-28f7-4719-8b31-a903874d800f" containerName="payload-processing" Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.214910 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc6e41e-117f-4521-a04b-3ccbb64a2d26" containerName="payload-pre-processing" Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.214915 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc6e41e-117f-4521-a04b-3ccbb64a2d26" containerName="payload-pre-processing" Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.214921 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f06fcb-d73f-4f53-b352-1d7f7a66566d" containerName="revoke-keys" Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.214927 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f06fcb-d73f-4f53-b352-1d7f7a66566d" containerName="revoke-keys" Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.215000 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbd13854-28f7-4719-8b31-a903874d800f" containerName="payload-processing" Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.215007 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc6e41e-117f-4521-a04b-3ccbb64a2d26" containerName="payload-pre-processing" Sep 11 20:06:54.216919 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.215013 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f06fcb-d73f-4f53-b352-1d7f7a66566d" containerName="revoke-keys" Sep 11 20:06:54.218122 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.218105 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" Sep 11 20:06:54.221180 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.221152 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 20:06:54.221180 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.221157 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 20:06:54.227348 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.227325 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc"] Sep 11 20:06:54.347644 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.347608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c973aa-dcd5-4a53-8245-04a69b06e0da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc\" (UID: \"23c973aa-dcd5-4a53-8245-04a69b06e0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" Sep 11 20:06:54.448254 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.448221 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c973aa-dcd5-4a53-8245-04a69b06e0da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc\" (UID: \"23c973aa-dcd5-4a53-8245-04a69b06e0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" Sep 11 20:06:54.448869 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.448849 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c973aa-dcd5-4a53-8245-04a69b06e0da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc\" (UID: \"23c973aa-dcd5-4a53-8245-04a69b06e0da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" Sep 11 20:06:54.530030 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.529986 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" Sep 11 20:06:54.808808 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:54.808781 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc"] Sep 11 20:06:54.810864 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:06:54.810832 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c973aa_dcd5_4a53_8245_04a69b06e0da.slice/crio-4d6b41ad4dd421ef1cc9d26ebd1d3dfd80bce764b21b84ce2e9cf5aeb51efe91 WatchSource:0}: Error finding container 4d6b41ad4dd421ef1cc9d26ebd1d3dfd80bce764b21b84ce2e9cf5aeb51efe91: Status 404 returned error can't find the container with id 4d6b41ad4dd421ef1cc9d26ebd1d3dfd80bce764b21b84ce2e9cf5aeb51efe91 Sep 11 20:06:55.398420 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:55.398327 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" event={"ID":"23c973aa-dcd5-4a53-8245-04a69b06e0da","Type":"ContainerStarted","Data":"16daea4fed588effcc74fc63695978d3e7b984bee8c02adf58ab089a5f87efc9"} Sep 11 20:06:55.398420 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:55.398395 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" event={"ID":"23c973aa-dcd5-4a53-8245-04a69b06e0da","Type":"ContainerStarted","Data":"4d6b41ad4dd421ef1cc9d26ebd1d3dfd80bce764b21b84ce2e9cf5aeb51efe91"} Sep 11 20:06:55.414486 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:55.414408 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" podStartSLOduration=1.414393686 podStartE2EDuration="1.414393686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:06:55.413353745 +0000 UTC m=+2448.685860945" watchObservedRunningTime="2026-09-11 20:06:55.414393686 +0000 UTC m=+2448.686900885" Sep 11 20:06:56.406386 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:56.406349 2559 generic.go:356] "Generic (PLEG): container finished" podID="23c973aa-dcd5-4a53-8245-04a69b06e0da" containerID="16daea4fed588effcc74fc63695978d3e7b984bee8c02adf58ab089a5f87efc9" exitCode=0 Sep 11 20:06:56.406782 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:56.406431 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" event={"ID":"23c973aa-dcd5-4a53-8245-04a69b06e0da","Type":"ContainerDied","Data":"16daea4fed588effcc74fc63695978d3e7b984bee8c02adf58ab089a5f87efc9"} Sep 11 20:06:57.549605 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:57.549576 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" Sep 11 20:06:57.581393 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:57.581368 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c973aa-dcd5-4a53-8245-04a69b06e0da-maas-api-service-ca\") pod \"23c973aa-dcd5-4a53-8245-04a69b06e0da\" (UID: \"23c973aa-dcd5-4a53-8245-04a69b06e0da\") " Sep 11 20:06:57.581782 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:57.581759 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c973aa-dcd5-4a53-8245-04a69b06e0da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23c973aa-dcd5-4a53-8245-04a69b06e0da" (UID: "23c973aa-dcd5-4a53-8245-04a69b06e0da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:06:57.682871 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:57.682789 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c973aa-dcd5-4a53-8245-04a69b06e0da-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:06:58.421882 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:58.421853 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" Sep 11 20:06:58.422060 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:58.421897 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc" event={"ID":"23c973aa-dcd5-4a53-8245-04a69b06e0da","Type":"ContainerDied","Data":"4d6b41ad4dd421ef1cc9d26ebd1d3dfd80bce764b21b84ce2e9cf5aeb51efe91"} Sep 11 20:06:58.422060 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:06:58.421940 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d6b41ad4dd421ef1cc9d26ebd1d3dfd80bce764b21b84ce2e9cf5aeb51efe91" Sep 11 20:07:04.906103 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:04.906064 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5"] Sep 11 20:07:04.906584 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:04.906315 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" podUID="fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3" containerName="payload-pre-processing" containerID="cri-o://e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d" gracePeriod=30 Sep 11 20:07:05.080616 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.080578 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc"] Sep 11 20:07:05.084565 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.084542 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df0d4-kpsrc"] Sep 11 20:07:05.162853 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.162786 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:07:05.224931 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.224891 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c973aa-dcd5-4a53-8245-04a69b06e0da" path="/var/lib/kubelet/pods/23c973aa-dcd5-4a53-8245-04a69b06e0da/volumes" Sep 11 20:07:05.256347 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.256312 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-plugins-config-volume\") pod \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\" (UID: \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\") " Sep 11 20:07:05.256504 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.256427 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g6mv\" (UniqueName: \"kubernetes.io/projected/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-kube-api-access-9g6mv\") pod \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\" (UID: \"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3\") " Sep 11 20:07:05.256686 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.256663 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3" (UID: "fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:05.256861 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.256847 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:05.258757 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.258725 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-kube-api-access-9g6mv" (OuterVolumeSpecName: "kube-api-access-9g6mv") pod "fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3" (UID: "fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3"). InnerVolumeSpecName "kube-api-access-9g6mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:05.357699 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.357646 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6mv\" (UniqueName: \"kubernetes.io/projected/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3-kube-api-access-9g6mv\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:05.464280 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.464193 2559 generic.go:356] "Generic (PLEG): container finished" podID="fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3" containerID="e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d" exitCode=0 Sep 11 20:07:05.464280 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.464240 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" Sep 11 20:07:05.464490 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.464286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" event={"ID":"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3","Type":"ContainerDied","Data":"e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d"} Sep 11 20:07:05.464490 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.464351 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5" event={"ID":"fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3","Type":"ContainerDied","Data":"21e0bd34369d7f299b8df344919015054f5349885cdbfef1e6d1a4c512e09629"} Sep 11 20:07:05.464490 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.464377 2559 scope.go:117] "RemoveContainer" containerID="e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d" Sep 11 20:07:05.474046 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.474028 2559 scope.go:117] "RemoveContainer" containerID="e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d" Sep 11 20:07:05.474352 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:07:05.474329 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d\": container with ID starting with e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d not found: ID does not exist" containerID="e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d" Sep 11 20:07:05.474403 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.474362 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d"} err="failed to get container status \"e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d\": rpc error: code = NotFound desc = could not find container \"e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d\": container with ID starting with e951c4a0ed6a8b378b84aca691db0159c60a13764e09278d2cd977317f32283d not found: ID does not exist" Sep 11 20:07:05.519286 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.519247 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5"] Sep 11 20:07:05.525027 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:05.524999 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5df0d4-5868d78fc8-jh7j5"] Sep 11 20:07:07.230721 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:07.230690 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3" path="/var/lib/kubelet/pods/fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3/volumes" Sep 11 20:07:08.262477 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:08.262382 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:07:12.023652 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.023620 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm"] Sep 11 20:07:12.024041 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.024027 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3" containerName="payload-pre-processing" Sep 11 20:07:12.024041 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.024039 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3" containerName="payload-pre-processing" Sep 11 20:07:12.024111 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.024053 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c973aa-dcd5-4a53-8245-04a69b06e0da" containerName="revoke-keys" Sep 11 20:07:12.024111 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.024058 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c973aa-dcd5-4a53-8245-04a69b06e0da" containerName="revoke-keys" Sep 11 20:07:12.024182 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.024116 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c973aa-dcd5-4a53-8245-04a69b06e0da" containerName="revoke-keys" Sep 11 20:07:12.024182 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.024125 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc5ed0aa-c5cf-4a6e-807b-4df2fe2799d3" containerName="payload-pre-processing" Sep 11 20:07:12.028618 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.028601 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" Sep 11 20:07:12.031754 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.031730 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 20:07:12.031918 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.031902 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 20:07:12.039028 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.039003 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm"] Sep 11 20:07:12.118908 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.118870 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a88a28-630d-45ba-8802-b0b4ed156b46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm\" (UID: \"c0a88a28-630d-45ba-8802-b0b4ed156b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" Sep 11 20:07:12.220331 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.220300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a88a28-630d-45ba-8802-b0b4ed156b46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm\" (UID: \"c0a88a28-630d-45ba-8802-b0b4ed156b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" Sep 11 20:07:12.220932 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.220911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a88a28-630d-45ba-8802-b0b4ed156b46-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm\" (UID: \"c0a88a28-630d-45ba-8802-b0b4ed156b46\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" Sep 11 20:07:12.341109 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.341028 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" Sep 11 20:07:12.620689 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:12.620658 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm"] Sep 11 20:07:12.622147 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:07:12.622117 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a88a28_630d_45ba_8802_b0b4ed156b46.slice/crio-195da3ff809d7f509c555072bd3fedc7b10aa09c38622d3a72bdc2cf10bb523c WatchSource:0}: Error finding container 195da3ff809d7f509c555072bd3fedc7b10aa09c38622d3a72bdc2cf10bb523c: Status 404 returned error can't find the container with id 195da3ff809d7f509c555072bd3fedc7b10aa09c38622d3a72bdc2cf10bb523c Sep 11 20:07:13.515333 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:13.515293 2559 generic.go:356] "Generic (PLEG): container finished" podID="c0a88a28-630d-45ba-8802-b0b4ed156b46" containerID="81954de9bd32254adac0052f76e8b1adb7085ab4e7a0487ce88c87a70743bae6" exitCode=0 Sep 11 20:07:13.515741 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:13.515339 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" event={"ID":"c0a88a28-630d-45ba-8802-b0b4ed156b46","Type":"ContainerDied","Data":"81954de9bd32254adac0052f76e8b1adb7085ab4e7a0487ce88c87a70743bae6"} Sep 11 20:07:13.515741 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:13.515391 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" event={"ID":"c0a88a28-630d-45ba-8802-b0b4ed156b46","Type":"ContainerStarted","Data":"195da3ff809d7f509c555072bd3fedc7b10aa09c38622d3a72bdc2cf10bb523c"} Sep 11 20:07:14.662057 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:14.662033 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" Sep 11 20:07:14.746299 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:14.746266 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a88a28-630d-45ba-8802-b0b4ed156b46-maas-api-service-ca\") pod \"c0a88a28-630d-45ba-8802-b0b4ed156b46\" (UID: \"c0a88a28-630d-45ba-8802-b0b4ed156b46\") " Sep 11 20:07:14.746644 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:14.746623 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0a88a28-630d-45ba-8802-b0b4ed156b46-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c0a88a28-630d-45ba-8802-b0b4ed156b46" (UID: "c0a88a28-630d-45ba-8802-b0b4ed156b46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:14.847347 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:14.847262 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a88a28-630d-45ba-8802-b0b4ed156b46-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:15.530104 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:15.530071 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" event={"ID":"c0a88a28-630d-45ba-8802-b0b4ed156b46","Type":"ContainerDied","Data":"195da3ff809d7f509c555072bd3fedc7b10aa09c38622d3a72bdc2cf10bb523c"} Sep 11 20:07:15.530104 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:15.530093 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm" Sep 11 20:07:15.530326 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:15.530121 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195da3ff809d7f509c555072bd3fedc7b10aa09c38622d3a72bdc2cf10bb523c" Sep 11 20:07:22.531445 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.531398 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc"] Sep 11 20:07:22.534831 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.534806 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-58c78c-29819280-mf9pc"] Sep 11 20:07:22.554743 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.554712 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh"] Sep 11 20:07:22.554987 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.554960 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" podUID="f13784bc-b8ce-47fb-8436-ffa68726db3b" containerName="payload-processing" containerID="cri-o://0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231" gracePeriod=30 Sep 11 20:07:22.566904 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.566874 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq"] Sep 11 20:07:22.567708 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.567679 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" podUID="f8fc3401-2495-4c9b-9a48-d2fec46767ee" containerName="payload-pre-processing" containerID="cri-o://b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3" gracePeriod=30 Sep 11 20:07:22.722870 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.722812 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm"] Sep 11 20:07:22.725322 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.725291 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-58c78c-hbmxm"] Sep 11 20:07:22.841620 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.841599 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 20:07:22.844250 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.844230 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 20:07:22.924279 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.924245 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9mpd\" (UniqueName: \"kubernetes.io/projected/f8fc3401-2495-4c9b-9a48-d2fec46767ee-kube-api-access-n9mpd\") pod \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\" (UID: \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\") " Sep 11 20:07:22.924494 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.924288 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc3401-2495-4c9b-9a48-d2fec46767ee-plugins-config-volume\") pod \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\" (UID: \"f8fc3401-2495-4c9b-9a48-d2fec46767ee\") " Sep 11 20:07:22.924494 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.924357 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13784bc-b8ce-47fb-8436-ffa68726db3b-plugins-config-volume\") pod \"f13784bc-b8ce-47fb-8436-ffa68726db3b\" (UID: \"f13784bc-b8ce-47fb-8436-ffa68726db3b\") " Sep 11 20:07:22.924494 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.924406 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcqwn\" (UniqueName: \"kubernetes.io/projected/f13784bc-b8ce-47fb-8436-ffa68726db3b-kube-api-access-dcqwn\") pod \"f13784bc-b8ce-47fb-8436-ffa68726db3b\" (UID: \"f13784bc-b8ce-47fb-8436-ffa68726db3b\") " Sep 11 20:07:22.924789 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.924764 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13784bc-b8ce-47fb-8436-ffa68726db3b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f13784bc-b8ce-47fb-8436-ffa68726db3b" (UID: "f13784bc-b8ce-47fb-8436-ffa68726db3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:22.924789 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.924774 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8fc3401-2495-4c9b-9a48-d2fec46767ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f8fc3401-2495-4c9b-9a48-d2fec46767ee" (UID: "f8fc3401-2495-4c9b-9a48-d2fec46767ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:22.926695 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.926674 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8fc3401-2495-4c9b-9a48-d2fec46767ee-kube-api-access-n9mpd" (OuterVolumeSpecName: "kube-api-access-n9mpd") pod "f8fc3401-2495-4c9b-9a48-d2fec46767ee" (UID: "f8fc3401-2495-4c9b-9a48-d2fec46767ee"). InnerVolumeSpecName "kube-api-access-n9mpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:22.926776 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:22.926759 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13784bc-b8ce-47fb-8436-ffa68726db3b-kube-api-access-dcqwn" (OuterVolumeSpecName: "kube-api-access-dcqwn") pod "f13784bc-b8ce-47fb-8436-ffa68726db3b" (UID: "f13784bc-b8ce-47fb-8436-ffa68726db3b"). InnerVolumeSpecName "kube-api-access-dcqwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:23.025630 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.025581 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f13784bc-b8ce-47fb-8436-ffa68726db3b-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:23.025630 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.025624 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcqwn\" (UniqueName: \"kubernetes.io/projected/f13784bc-b8ce-47fb-8436-ffa68726db3b-kube-api-access-dcqwn\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:23.025630 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.025635 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mpd\" (UniqueName: \"kubernetes.io/projected/f8fc3401-2495-4c9b-9a48-d2fec46767ee-kube-api-access-n9mpd\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:23.025925 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.025664 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8fc3401-2495-4c9b-9a48-d2fec46767ee-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:23.230302 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.230220 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42f74184-da9b-4932-9603-b46696237ae7" path="/var/lib/kubelet/pods/42f74184-da9b-4932-9603-b46696237ae7/volumes" Sep 11 20:07:23.230558 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.230542 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0a88a28-630d-45ba-8802-b0b4ed156b46" path="/var/lib/kubelet/pods/c0a88a28-630d-45ba-8802-b0b4ed156b46/volumes" Sep 11 20:07:23.593783 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.593751 2559 generic.go:356] "Generic (PLEG): container finished" podID="f13784bc-b8ce-47fb-8436-ffa68726db3b" containerID="0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231" exitCode=0 Sep 11 20:07:23.594208 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.593826 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" Sep 11 20:07:23.594208 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.593843 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" event={"ID":"f13784bc-b8ce-47fb-8436-ffa68726db3b","Type":"ContainerDied","Data":"0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231"} Sep 11 20:07:23.594208 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.593906 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh" event={"ID":"f13784bc-b8ce-47fb-8436-ffa68726db3b","Type":"ContainerDied","Data":"129d8a01500d8e89b833ba14891172346eebdb643b229effc0fa2c6e3e697401"} Sep 11 20:07:23.594208 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.593934 2559 scope.go:117] "RemoveContainer" containerID="0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231" Sep 11 20:07:23.595522 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.595286 2559 generic.go:356] "Generic (PLEG): container finished" podID="f8fc3401-2495-4c9b-9a48-d2fec46767ee" containerID="b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3" exitCode=0 Sep 11 20:07:23.595522 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.595341 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" event={"ID":"f8fc3401-2495-4c9b-9a48-d2fec46767ee","Type":"ContainerDied","Data":"b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3"} Sep 11 20:07:23.595522 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.595361 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" Sep 11 20:07:23.595522 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.595390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq" event={"ID":"f8fc3401-2495-4c9b-9a48-d2fec46767ee","Type":"ContainerDied","Data":"e047beb674d5a61329db32c0fd912e42699b8c7e8e299976b12d3d7982c16e29"} Sep 11 20:07:23.604342 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.604324 2559 scope.go:117] "RemoveContainer" containerID="0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231" Sep 11 20:07:23.604690 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:07:23.604662 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231\": container with ID starting with 0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231 not found: ID does not exist" containerID="0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231" Sep 11 20:07:23.604780 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.604698 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231"} err="failed to get container status \"0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231\": rpc error: code = NotFound desc = could not find container \"0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231\": container with ID starting with 0b9a6ac08404327292a33c24970044068797c51cc53ca422a4ebc0f26cc08231 not found: ID does not exist" Sep 11 20:07:23.604780 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.604716 2559 scope.go:117] "RemoveContainer" containerID="b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3" Sep 11 20:07:23.614174 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.614155 2559 scope.go:117] "RemoveContainer" containerID="b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3" Sep 11 20:07:23.614481 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:07:23.614430 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3\": container with ID starting with b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3 not found: ID does not exist" containerID="b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3" Sep 11 20:07:23.614562 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.614490 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3"} err="failed to get container status \"b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3\": rpc error: code = NotFound desc = could not find container \"b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3\": container with ID starting with b5b99f0590c82d3e7c37835de48ce7cb1d61496785880edaec60ca2f338d3dc3 not found: ID does not exist" Sep 11 20:07:23.614665 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.614647 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh"] Sep 11 20:07:23.618648 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.618629 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-58c78c-bcfb46c95-xn8kh"] Sep 11 20:07:23.628054 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.628031 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq"] Sep 11 20:07:23.634755 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:23.634735 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-58c78c-b79b76646-574kq"] Sep 11 20:07:25.224037 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:25.223992 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f13784bc-b8ce-47fb-8436-ffa68726db3b" path="/var/lib/kubelet/pods/f13784bc-b8ce-47fb-8436-ffa68726db3b/volumes" Sep 11 20:07:25.224571 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:25.224552 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8fc3401-2495-4c9b-9a48-d2fec46767ee" path="/var/lib/kubelet/pods/f8fc3401-2495-4c9b-9a48-d2fec46767ee/volumes" Sep 11 20:07:25.455749 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:25.455713 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:07:29.725386 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725347 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg"] Sep 11 20:07:29.725812 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725737 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0a88a28-630d-45ba-8802-b0b4ed156b46" containerName="revoke-keys" Sep 11 20:07:29.725812 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725748 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0a88a28-630d-45ba-8802-b0b4ed156b46" containerName="revoke-keys" Sep 11 20:07:29.725812 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725763 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8fc3401-2495-4c9b-9a48-d2fec46767ee" containerName="payload-pre-processing" Sep 11 20:07:29.725812 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725768 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8fc3401-2495-4c9b-9a48-d2fec46767ee" containerName="payload-pre-processing" Sep 11 20:07:29.725812 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725774 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f13784bc-b8ce-47fb-8436-ffa68726db3b" containerName="payload-processing" Sep 11 20:07:29.725812 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725780 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f13784bc-b8ce-47fb-8436-ffa68726db3b" containerName="payload-processing" Sep 11 20:07:29.726007 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725856 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8fc3401-2495-4c9b-9a48-d2fec46767ee" containerName="payload-pre-processing" Sep 11 20:07:29.726007 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725865 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0a88a28-630d-45ba-8802-b0b4ed156b46" containerName="revoke-keys" Sep 11 20:07:29.726007 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.725873 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f13784bc-b8ce-47fb-8436-ffa68726db3b" containerName="payload-processing" Sep 11 20:07:29.730359 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.730342 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" Sep 11 20:07:29.734769 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.734748 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 20:07:29.734892 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.734747 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgbxs\"" Sep 11 20:07:29.747011 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.746990 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg"] Sep 11 20:07:29.887291 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.887257 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg\" (UID: \"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" Sep 11 20:07:29.988715 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.988679 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg\" (UID: \"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" Sep 11 20:07:29.989356 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:29.989337 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg\" (UID: \"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" Sep 11 20:07:30.040278 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:30.040246 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" Sep 11 20:07:30.315276 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:30.315250 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg"] Sep 11 20:07:30.317505 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:07:30.317476 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a819d2_ee4d_487e_9ce4_8a6562e5f8f1.slice/crio-28ee1f1f4cee92464a3072e9caaebb4f58d1bb9c4f5b1c1ac0c99a407fd2e7cd WatchSource:0}: Error finding container 28ee1f1f4cee92464a3072e9caaebb4f58d1bb9c4f5b1c1ac0c99a407fd2e7cd: Status 404 returned error can't find the container with id 28ee1f1f4cee92464a3072e9caaebb4f58d1bb9c4f5b1c1ac0c99a407fd2e7cd Sep 11 20:07:30.643522 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:30.643477 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" event={"ID":"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1","Type":"ContainerStarted","Data":"ea2bf624e7a27ef5db47f4bec868159ddf612904af5d58ab6207b93aeea39b96"} Sep 11 20:07:30.643676 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:30.643531 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" event={"ID":"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1","Type":"ContainerStarted","Data":"28ee1f1f4cee92464a3072e9caaebb4f58d1bb9c4f5b1c1ac0c99a407fd2e7cd"} Sep 11 20:07:30.661241 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:30.661194 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" podStartSLOduration=1.661177623 podStartE2EDuration="1.661177623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:07:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:07:30.659629095 +0000 UTC m=+2483.932136295" watchObservedRunningTime="2026-09-11 20:07:30.661177623 +0000 UTC m=+2483.933684814" Sep 11 20:07:31.650929 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:31.650893 2559 generic.go:356] "Generic (PLEG): container finished" podID="e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1" containerID="ea2bf624e7a27ef5db47f4bec868159ddf612904af5d58ab6207b93aeea39b96" exitCode=0 Sep 11 20:07:31.651357 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:31.650972 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" event={"ID":"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1","Type":"ContainerDied","Data":"ea2bf624e7a27ef5db47f4bec868159ddf612904af5d58ab6207b93aeea39b96"} Sep 11 20:07:32.790025 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:32.789997 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" Sep 11 20:07:32.918542 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:32.918419 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1-maas-api-service-ca\") pod \"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1\" (UID: \"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1\") " Sep 11 20:07:32.918801 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:32.918777 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1" (UID: "e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:33.020048 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:33.020015 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1-maas-api-service-ca\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:33.663410 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:33.663377 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" Sep 11 20:07:33.663410 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:33.663385 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg" event={"ID":"e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1","Type":"ContainerDied","Data":"28ee1f1f4cee92464a3072e9caaebb4f58d1bb9c4f5b1c1ac0c99a407fd2e7cd"} Sep 11 20:07:33.663643 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:33.663455 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ee1f1f4cee92464a3072e9caaebb4f58d1bb9c4f5b1c1ac0c99a407fd2e7cd" Sep 11 20:07:40.337150 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.337107 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh"] Sep 11 20:07:40.340320 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.340294 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae268e-29819280-w6gdh"] Sep 11 20:07:40.365896 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.365864 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv"] Sep 11 20:07:40.366132 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.366107 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" podUID="91e9132a-feec-4559-821f-ebfe74cd17cf" containerName="payload-processing" containerID="cri-o://78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd" gracePeriod=30 Sep 11 20:07:40.375610 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.375586 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs"] Sep 11 20:07:40.375782 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.375763 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" podUID="f7d7e2bd-4078-4152-ba9a-5984eca3ac9d" containerName="payload-pre-processing" containerID="cri-o://4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a" gracePeriod=30 Sep 11 20:07:40.551807 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.551751 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg"] Sep 11 20:07:40.554509 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.554487 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae268e-lmlfg"] Sep 11 20:07:40.626331 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.626305 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 20:07:40.656140 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.656113 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 20:07:40.686115 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.686087 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9132a-feec-4559-821f-ebfe74cd17cf-plugins-config-volume\") pod \"91e9132a-feec-4559-821f-ebfe74cd17cf\" (UID: \"91e9132a-feec-4559-821f-ebfe74cd17cf\") " Sep 11 20:07:40.686251 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.686150 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/91e9132a-feec-4559-821f-ebfe74cd17cf-kube-api-access-l9qth\") pod \"91e9132a-feec-4559-821f-ebfe74cd17cf\" (UID: \"91e9132a-feec-4559-821f-ebfe74cd17cf\") " Sep 11 20:07:40.686251 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.686222 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2fr2\" (UniqueName: \"kubernetes.io/projected/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-kube-api-access-p2fr2\") pod \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\" (UID: \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\") " Sep 11 20:07:40.686389 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.686263 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-plugins-config-volume\") pod \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\" (UID: \"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d\") " Sep 11 20:07:40.686489 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.686461 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91e9132a-feec-4559-821f-ebfe74cd17cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "91e9132a-feec-4559-821f-ebfe74cd17cf" (UID: "91e9132a-feec-4559-821f-ebfe74cd17cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:40.686733 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.686710 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7d7e2bd-4078-4152-ba9a-5984eca3ac9d" (UID: "f7d7e2bd-4078-4152-ba9a-5984eca3ac9d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 20:07:40.686733 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.686725 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91e9132a-feec-4559-821f-ebfe74cd17cf-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:40.688430 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.688412 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e9132a-feec-4559-821f-ebfe74cd17cf-kube-api-access-l9qth" (OuterVolumeSpecName: "kube-api-access-l9qth") pod "91e9132a-feec-4559-821f-ebfe74cd17cf" (UID: "91e9132a-feec-4559-821f-ebfe74cd17cf"). InnerVolumeSpecName "kube-api-access-l9qth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:40.688535 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.688423 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-kube-api-access-p2fr2" (OuterVolumeSpecName: "kube-api-access-p2fr2") pod "f7d7e2bd-4078-4152-ba9a-5984eca3ac9d" (UID: "f7d7e2bd-4078-4152-ba9a-5984eca3ac9d"). InnerVolumeSpecName "kube-api-access-p2fr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:07:40.706432 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.706404 2559 generic.go:356] "Generic (PLEG): container finished" podID="f7d7e2bd-4078-4152-ba9a-5984eca3ac9d" containerID="4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a" exitCode=0 Sep 11 20:07:40.706561 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.706470 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" Sep 11 20:07:40.706561 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.706469 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" event={"ID":"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d","Type":"ContainerDied","Data":"4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a"} Sep 11 20:07:40.706561 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.706527 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs" event={"ID":"f7d7e2bd-4078-4152-ba9a-5984eca3ac9d","Type":"ContainerDied","Data":"e8be189070d8276ad39d55904345226a9557fde064cb283cfdeaf6ccff6995ae"} Sep 11 20:07:40.706561 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.706561 2559 scope.go:117] "RemoveContainer" containerID="4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a" Sep 11 20:07:40.708114 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.708094 2559 generic.go:356] "Generic (PLEG): container finished" podID="91e9132a-feec-4559-821f-ebfe74cd17cf" containerID="78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd" exitCode=0 Sep 11 20:07:40.708216 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.708186 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" event={"ID":"91e9132a-feec-4559-821f-ebfe74cd17cf","Type":"ContainerDied","Data":"78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd"} Sep 11 20:07:40.708283 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.708230 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" event={"ID":"91e9132a-feec-4559-821f-ebfe74cd17cf","Type":"ContainerDied","Data":"d5dd47bec93f028c253f2ae575d887fd3ba897da00496285ee845c478377a6e9"} Sep 11 20:07:40.708345 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.708297 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv" Sep 11 20:07:40.718496 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.718466 2559 scope.go:117] "RemoveContainer" containerID="4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a" Sep 11 20:07:40.718783 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:07:40.718761 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a\": container with ID starting with 4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a not found: ID does not exist" containerID="4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a" Sep 11 20:07:40.718868 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.718789 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a"} err="failed to get container status \"4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a\": rpc error: code = NotFound desc = could not find container \"4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a\": container with ID starting with 4dadc74e90a72a110a56d38de7d1b03d1d4ec8ef4391648fb6cdca2b7db3a61a not found: ID does not exist" Sep 11 20:07:40.718868 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.718804 2559 scope.go:117] "RemoveContainer" containerID="78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd" Sep 11 20:07:40.727319 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.727303 2559 scope.go:117] "RemoveContainer" containerID="78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd" Sep 11 20:07:40.727682 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:07:40.727657 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd\": container with ID starting with 78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd not found: ID does not exist" containerID="78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd" Sep 11 20:07:40.727794 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.727709 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd"} err="failed to get container status \"78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd\": rpc error: code = NotFound desc = could not find container \"78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd\": container with ID starting with 78003e61c59d79b7d57a5088177571e5871d0c9a9781f1dff52d87d3b0bf7bfd not found: ID does not exist" Sep 11 20:07:40.730624 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.730603 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs"] Sep 11 20:07:40.734113 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.734095 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae268e-5454fc7c9d-7vhvs"] Sep 11 20:07:40.744796 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.744776 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv"] Sep 11 20:07:40.748534 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.748516 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae268e-5549c49f67-hg5cv"] Sep 11 20:07:40.787930 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.787903 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2fr2\" (UniqueName: \"kubernetes.io/projected/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-kube-api-access-p2fr2\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:40.787930 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.787927 2559 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d-plugins-config-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:40.788085 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:40.787939 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9qth\" (UniqueName: \"kubernetes.io/projected/91e9132a-feec-4559-821f-ebfe74cd17cf-kube-api-access-l9qth\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:07:41.223880 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:41.223846 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91e9132a-feec-4559-821f-ebfe74cd17cf" path="/var/lib/kubelet/pods/91e9132a-feec-4559-821f-ebfe74cd17cf/volumes" Sep 11 20:07:41.224134 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:41.224122 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0f9a898-14fe-4d54-9ddf-597d6b0743b3" path="/var/lib/kubelet/pods/d0f9a898-14fe-4d54-9ddf-597d6b0743b3/volumes" Sep 11 20:07:41.224375 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:41.224364 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1" path="/var/lib/kubelet/pods/e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1/volumes" Sep 11 20:07:41.224691 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:41.224680 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7d7e2bd-4078-4152-ba9a-5984eca3ac9d" path="/var/lib/kubelet/pods/f7d7e2bd-4078-4152-ba9a-5984eca3ac9d/volumes" Sep 11 20:07:43.156458 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:43.156410 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:07:50.850522 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:07:50.850484 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:08:07.996317 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:08:07.996284 2559 scope.go:117] "RemoveContainer" containerID="cb59003a09b4be3663507ac4f793fae2b9db76c39f3e002e8d149d5c88501dc1" Sep 11 20:08:08.005150 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:08:08.005129 2559 scope.go:117] "RemoveContainer" containerID="deb1a36e1b6c8c0baff85efb47da616a5394342da673679463e33a0d1c2e5958" Sep 11 20:09:11.857699 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:09:11.857666 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:09:36.471304 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:09:36.471218 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:09:44.357284 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:09:44.357247 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:09:52.362389 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:09:52.362355 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:10:10.159704 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:10.159668 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:10:18.361301 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:18.361263 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:10:19.494227 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:19.494191 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg"] Sep 11 20:10:19.494721 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:19.494395 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" podUID="480d0f51-3b90-4927-9ab9-258632a66d19" containerName="manager" containerID="cri-o://c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1" gracePeriod=10 Sep 11 20:10:21.245168 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.245142 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 20:10:21.349647 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.349614 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4ql4\" (UniqueName: \"kubernetes.io/projected/480d0f51-3b90-4927-9ab9-258632a66d19-kube-api-access-w4ql4\") pod \"480d0f51-3b90-4927-9ab9-258632a66d19\" (UID: \"480d0f51-3b90-4927-9ab9-258632a66d19\") " Sep 11 20:10:21.349861 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.349714 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/480d0f51-3b90-4927-9ab9-258632a66d19-extensions-socket-volume\") pod \"480d0f51-3b90-4927-9ab9-258632a66d19\" (UID: \"480d0f51-3b90-4927-9ab9-258632a66d19\") " Sep 11 20:10:21.350074 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.350048 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480d0f51-3b90-4927-9ab9-258632a66d19-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "480d0f51-3b90-4927-9ab9-258632a66d19" (UID: "480d0f51-3b90-4927-9ab9-258632a66d19"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 20:10:21.352028 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.352002 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480d0f51-3b90-4927-9ab9-258632a66d19-kube-api-access-w4ql4" (OuterVolumeSpecName: "kube-api-access-w4ql4") pod "480d0f51-3b90-4927-9ab9-258632a66d19" (UID: "480d0f51-3b90-4927-9ab9-258632a66d19"). InnerVolumeSpecName "kube-api-access-w4ql4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 20:10:21.450755 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.450634 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/480d0f51-3b90-4927-9ab9-258632a66d19-extensions-socket-volume\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:10:21.450755 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.450701 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4ql4\" (UniqueName: \"kubernetes.io/projected/480d0f51-3b90-4927-9ab9-258632a66d19-kube-api-access-w4ql4\") on node \"ip-10-0-137-97.ec2.internal\" DevicePath \"\"" Sep 11 20:10:21.673045 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.673010 2559 generic.go:356] "Generic (PLEG): container finished" podID="480d0f51-3b90-4927-9ab9-258632a66d19" containerID="c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1" exitCode=0 Sep 11 20:10:21.673244 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.673072 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" Sep 11 20:10:21.673244 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.673102 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" event={"ID":"480d0f51-3b90-4927-9ab9-258632a66d19","Type":"ContainerDied","Data":"c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1"} Sep 11 20:10:21.673244 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.673192 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg" event={"ID":"480d0f51-3b90-4927-9ab9-258632a66d19","Type":"ContainerDied","Data":"050a4880f14560a5bcfa5668e8199221250979be206f53fd55ba55f5b9dd8340"} Sep 11 20:10:21.673244 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.673223 2559 scope.go:117] "RemoveContainer" containerID="c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1" Sep 11 20:10:21.683294 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.683277 2559 scope.go:117] "RemoveContainer" containerID="c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1" Sep 11 20:10:21.683626 ip-10-0-137-97 kubenswrapper[2559]: E0911 20:10:21.683606 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1\": container with ID starting with c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1 not found: ID does not exist" containerID="c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1" Sep 11 20:10:21.683706 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.683638 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1"} err="failed to get container status \"c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1\": rpc error: code = NotFound desc = could not find container \"c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1\": container with ID starting with c0c26f2a24aded46428a556b99ef666c90d1df83c84b2573fe240b623e5be9a1 not found: ID does not exist" Sep 11 20:10:21.696765 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.696741 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg"] Sep 11 20:10:21.700172 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:21.700151 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-dzzpg"] Sep 11 20:10:23.224398 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:10:23.224365 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="480d0f51-3b90-4927-9ab9-258632a66d19" path="/var/lib/kubelet/pods/480d0f51-3b90-4927-9ab9-258632a66d19/volumes" Sep 11 20:11:20.176526 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176488 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj"] Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176872 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="480d0f51-3b90-4927-9ab9-258632a66d19" containerName="manager" Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176885 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="480d0f51-3b90-4927-9ab9-258632a66d19" containerName="manager" Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176896 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91e9132a-feec-4559-821f-ebfe74cd17cf" containerName="payload-processing" Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176902 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="91e9132a-feec-4559-821f-ebfe74cd17cf" containerName="payload-processing" Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176920 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7d7e2bd-4078-4152-ba9a-5984eca3ac9d" containerName="payload-pre-processing" Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176926 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7d7e2bd-4078-4152-ba9a-5984eca3ac9d" containerName="payload-pre-processing" Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176938 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1" containerName="revoke-keys" Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176943 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1" containerName="revoke-keys" Sep 11 20:11:20.177002 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.176999 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7d7e2bd-4078-4152-ba9a-5984eca3ac9d" containerName="payload-pre-processing" Sep 11 20:11:20.177280 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.177009 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="91e9132a-feec-4559-821f-ebfe74cd17cf" containerName="payload-processing" Sep 11 20:11:20.177280 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.177016 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5a819d2-ee4d-487e-9ce4-8a6562e5f8f1" containerName="revoke-keys" Sep 11 20:11:20.177280 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.177022 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="480d0f51-3b90-4927-9ab9-258632a66d19" containerName="manager" Sep 11 20:11:20.180111 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.180093 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:20.183117 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.183089 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5xldn\"" Sep 11 20:11:20.193114 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.193085 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj"] Sep 11 20:11:20.197823 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.197789 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk9gf\" (UniqueName: \"kubernetes.io/projected/2691a0c5-5b67-440c-923f-046ddacdf519-kube-api-access-bk9gf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rnthj\" (UID: \"2691a0c5-5b67-440c-923f-046ddacdf519\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:20.198001 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.197981 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2691a0c5-5b67-440c-923f-046ddacdf519-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rnthj\" (UID: \"2691a0c5-5b67-440c-923f-046ddacdf519\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:20.298719 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.298678 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2691a0c5-5b67-440c-923f-046ddacdf519-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rnthj\" (UID: \"2691a0c5-5b67-440c-923f-046ddacdf519\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:20.298921 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.298829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk9gf\" (UniqueName: \"kubernetes.io/projected/2691a0c5-5b67-440c-923f-046ddacdf519-kube-api-access-bk9gf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rnthj\" (UID: \"2691a0c5-5b67-440c-923f-046ddacdf519\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:20.299070 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.299046 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2691a0c5-5b67-440c-923f-046ddacdf519-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rnthj\" (UID: \"2691a0c5-5b67-440c-923f-046ddacdf519\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:20.307579 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.307547 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk9gf\" (UniqueName: \"kubernetes.io/projected/2691a0c5-5b67-440c-923f-046ddacdf519-kube-api-access-bk9gf\") pod \"kuadrant-operator-controller-manager-5769f99d9f-rnthj\" (UID: \"2691a0c5-5b67-440c-923f-046ddacdf519\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:20.490612 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.490581 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:20.776732 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:20.776700 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj"] Sep 11 20:11:20.777652 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:11:20.777629 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2691a0c5_5b67_440c_923f_046ddacdf519.slice/crio-f4097c9ceb3671ed45a089133ed519f76316115383a81ce33288508bfb4809bf WatchSource:0}: Error finding container f4097c9ceb3671ed45a089133ed519f76316115383a81ce33288508bfb4809bf: Status 404 returned error can't find the container with id f4097c9ceb3671ed45a089133ed519f76316115383a81ce33288508bfb4809bf Sep 11 20:11:21.022340 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:21.022304 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" event={"ID":"2691a0c5-5b67-440c-923f-046ddacdf519","Type":"ContainerStarted","Data":"e848578f72e77a5ad449e933b01d279d54d68409fc9f7439ceae88d78df46af6"} Sep 11 20:11:21.022554 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:21.022368 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" event={"ID":"2691a0c5-5b67-440c-923f-046ddacdf519","Type":"ContainerStarted","Data":"f4097c9ceb3671ed45a089133ed519f76316115383a81ce33288508bfb4809bf"} Sep 11 20:11:21.022554 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:21.022410 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:21.043719 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:21.043614 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" podStartSLOduration=1.043597772 podStartE2EDuration="1.043597772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:11:21.042135308 +0000 UTC m=+2714.314642508" watchObservedRunningTime="2026-09-11 20:11:21.043597772 +0000 UTC m=+2714.316104972" Sep 11 20:11:32.030896 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:32.030859 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-rnthj" Sep 11 20:11:43.856746 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:43.856654 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:11:51.747839 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:51.747793 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:11:59.657375 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:11:59.657333 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:12:08.074233 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:12:08.074200 2559 scope.go:117] "RemoveContainer" containerID="6e2c57f7df725113eed4d6a67439a1a39e91232d4095db371613d6ac64d2f572" Sep 11 20:12:08.083005 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:12:08.082983 2559 scope.go:117] "RemoveContainer" containerID="773dc9add8afe8e3557672fbe611ee661c52bebce5c4b903b578d640dcca7e2a" Sep 11 20:12:30.652021 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:12:30.651941 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:12:46.859279 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:12:46.859237 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:12:55.568776 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:12:55.568738 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:13:03.061821 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:03.061788 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:13:08.108695 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:08.108659 2559 scope.go:117] "RemoveContainer" containerID="16daea4fed588effcc74fc63695978d3e7b984bee8c02adf58ab089a5f87efc9" Sep 11 20:13:09.362619 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:09.362580 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:13:16.966645 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:16.966600 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:13:23.966499 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:23.966462 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:13:29.863118 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:29.863080 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:13:36.561827 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:36.561789 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-srsnl"] Sep 11 20:13:39.367984 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:39.367945 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4d6bd67f-545jz_9d68df59-dab0-4bf2-bb4e-80f28f7797cb/authorino/0.log" Sep 11 20:13:44.744450 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:44.744387 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-88bmh_1e2c2640-d759-41bc-bc41-7f57b39cb976/manager/1.log" Sep 11 20:13:45.207261 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:45.207181 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-kgkdj_d4bffeba-f3e3-49a1-95c6-31b4a31d0755/copy-manifests/0.log" Sep 11 20:13:45.213032 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:45.213006 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-kgkdj_d4bffeba-f3e3-49a1-95c6-31b4a31d0755/manager/1.log" Sep 11 20:13:46.047177 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:46.047145 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29819265-hpn2w_cc858de9-7553-403d-82dc-9d9a16d971f7/cleanup/0.log" Sep 11 20:13:47.027075 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:47.027042 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4d6bd67f-545jz_9d68df59-dab0-4bf2-bb4e-80f28f7797cb/authorino/0.log" Sep 11 20:13:47.141597 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:47.141563 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fjhz5_274c0e01-91ab-430b-8abc-ce22883d0705/manager/0.log" Sep 11 20:13:47.355282 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:47.355209 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-qrst9_ec4b84da-6719-4343-9536-b324581bac1c/kuadrant-console-plugin/0.log" Sep 11 20:13:47.466991 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:47.466963 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-rnthj_2691a0c5-5b67-440c-923f-046ddacdf519/manager/0.log" Sep 11 20:13:47.591321 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:47.591288 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-srsnl_90c02666-75a6-48b9-9497-5e098f391ce5/limitador/0.log" Sep 11 20:13:47.713320 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:47.713249 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-prh59_2818e853-46df-4e26-9c31-0f77fdf9c5c8/manager/0.log" Sep 11 20:13:48.377106 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:48.377078 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-596df76fbd-kls4w_112e02c3-a598-4433-b801-d748a0a949f9/kube-auth-proxy/0.log" Sep 11 20:13:49.402319 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:49.402285 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-mm62p_d6b1b20d-7420-42b1-a5b0-ed4bf18d08a1/main/0.log" Sep 11 20:13:49.510264 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:49.510239 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-7rj45_10aed8d0-1812-452b-8bea-013d57318766/main/0.log" Sep 11 20:13:49.837350 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:49.837318 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-7cn6n_5213e0e1-9121-4c72-a026-4f280dda8a5c/main/0.log" Sep 11 20:13:54.655324 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.655285 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kx729/must-gather-vjg85"] Sep 11 20:13:54.658926 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.658907 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kx729/must-gather-vjg85" Sep 11 20:13:54.661685 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.661666 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kx729\"/\"kube-root-ca.crt\"" Sep 11 20:13:54.661949 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.661932 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kx729\"/\"default-dockercfg-hfjpd\"" Sep 11 20:13:54.661949 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.661943 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kx729\"/\"openshift-service-ca.crt\"" Sep 11 20:13:54.669816 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.669792 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kx729/must-gather-vjg85"] Sep 11 20:13:54.690221 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.690194 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmdp9\" (UniqueName: \"kubernetes.io/projected/c1a9765c-7f34-47ba-8542-e2bd36dd987f-kube-api-access-vmdp9\") pod \"must-gather-vjg85\" (UID: \"c1a9765c-7f34-47ba-8542-e2bd36dd987f\") " pod="openshift-must-gather-kx729/must-gather-vjg85" Sep 11 20:13:54.690346 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.690271 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1a9765c-7f34-47ba-8542-e2bd36dd987f-must-gather-output\") pod \"must-gather-vjg85\" (UID: \"c1a9765c-7f34-47ba-8542-e2bd36dd987f\") " pod="openshift-must-gather-kx729/must-gather-vjg85" Sep 11 20:13:54.791247 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.791217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmdp9\" (UniqueName: \"kubernetes.io/projected/c1a9765c-7f34-47ba-8542-e2bd36dd987f-kube-api-access-vmdp9\") pod \"must-gather-vjg85\" (UID: \"c1a9765c-7f34-47ba-8542-e2bd36dd987f\") " pod="openshift-must-gather-kx729/must-gather-vjg85" Sep 11 20:13:54.791426 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.791281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1a9765c-7f34-47ba-8542-e2bd36dd987f-must-gather-output\") pod \"must-gather-vjg85\" (UID: \"c1a9765c-7f34-47ba-8542-e2bd36dd987f\") " pod="openshift-must-gather-kx729/must-gather-vjg85" Sep 11 20:13:54.791606 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.791589 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c1a9765c-7f34-47ba-8542-e2bd36dd987f-must-gather-output\") pod \"must-gather-vjg85\" (UID: \"c1a9765c-7f34-47ba-8542-e2bd36dd987f\") " pod="openshift-must-gather-kx729/must-gather-vjg85" Sep 11 20:13:54.800178 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.800155 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmdp9\" (UniqueName: \"kubernetes.io/projected/c1a9765c-7f34-47ba-8542-e2bd36dd987f-kube-api-access-vmdp9\") pod \"must-gather-vjg85\" (UID: \"c1a9765c-7f34-47ba-8542-e2bd36dd987f\") " pod="openshift-must-gather-kx729/must-gather-vjg85" Sep 11 20:13:54.968599 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:54.968513 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kx729/must-gather-vjg85" Sep 11 20:13:55.244666 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:55.244629 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kx729/must-gather-vjg85"] Sep 11 20:13:55.251197 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:13:55.251171 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1a9765c_7f34_47ba_8542_e2bd36dd987f.slice/crio-a46cba78c1d1412f55c32e0844d097c748989cd22747670c13f914f38e2e01b2 WatchSource:0}: Error finding container a46cba78c1d1412f55c32e0844d097c748989cd22747670c13f914f38e2e01b2: Status 404 returned error can't find the container with id a46cba78c1d1412f55c32e0844d097c748989cd22747670c13f914f38e2e01b2 Sep 11 20:13:55.253137 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:55.253118 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 20:13:55.924933 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:55.924887 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kx729/must-gather-vjg85" event={"ID":"c1a9765c-7f34-47ba-8542-e2bd36dd987f","Type":"ContainerStarted","Data":"a46cba78c1d1412f55c32e0844d097c748989cd22747670c13f914f38e2e01b2"} Sep 11 20:13:56.936749 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:56.936701 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kx729/must-gather-vjg85" event={"ID":"c1a9765c-7f34-47ba-8542-e2bd36dd987f","Type":"ContainerStarted","Data":"42a9fcf8cbb330ae06e6c0962aa5e80ea0ba472f95b55b88e3a9207509e4895a"} Sep 11 20:13:56.937229 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:56.936769 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kx729/must-gather-vjg85" event={"ID":"c1a9765c-7f34-47ba-8542-e2bd36dd987f","Type":"ContainerStarted","Data":"55f13ecb25b020cf83f6d7276da88072e35eb253f001e54e2033dd2261d028f3"} Sep 11 20:13:56.953152 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:56.953095 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kx729/must-gather-vjg85" podStartSLOduration=2.076216942 podStartE2EDuration="2.953079221s" totalImagesPullingTime="876.862279ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 20:13:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:13:56.951663674 +0000 UTC m=+2870.224170878" watchObservedRunningTime="2026-09-11 20:13:56.953079221 +0000 UTC m=+2870.225586421" Sep 11 20:13:57.741192 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:57.741160 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dbf4v_321c2253-fa90-47e2-84b8-10400adb7630/global-pull-secret-syncer/0.log" Sep 11 20:13:57.932500 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:57.932471 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zv4k9_c478c692-0fc0-4dd7-8e9a-4360e4c73643/konnectivity-agent/0.log" Sep 11 20:13:57.974790 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:13:57.974758 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-97.ec2.internal_9a03fbe7e7fc68e01ff8c561ad56ba60/haproxy/0.log" Sep 11 20:14:02.690747 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:02.688743 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-b4d6bd67f-545jz_9d68df59-dab0-4bf2-bb4e-80f28f7797cb/authorino/0.log" Sep 11 20:14:02.718216 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:02.718168 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-fjhz5_274c0e01-91ab-430b-8abc-ce22883d0705/manager/0.log" Sep 11 20:14:02.776091 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:02.776046 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-qrst9_ec4b84da-6719-4343-9536-b324581bac1c/kuadrant-console-plugin/0.log" Sep 11 20:14:02.818592 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:02.818559 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-rnthj_2691a0c5-5b67-440c-923f-046ddacdf519/manager/0.log" Sep 11 20:14:02.839821 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:02.839790 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-srsnl_90c02666-75a6-48b9-9497-5e098f391ce5/limitador/0.log" Sep 11 20:14:02.915368 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:02.915260 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-prh59_2818e853-46df-4e26-9c31-0f77fdf9c5c8/manager/0.log" Sep 11 20:14:04.430625 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.430593 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da9c672e-9851-4719-8809-d54d98d1dd5b/alertmanager/0.log" Sep 11 20:14:04.457543 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.457496 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da9c672e-9851-4719-8809-d54d98d1dd5b/config-reloader/0.log" Sep 11 20:14:04.486000 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.485972 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da9c672e-9851-4719-8809-d54d98d1dd5b/kube-rbac-proxy-web/0.log" Sep 11 20:14:04.508726 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.508686 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da9c672e-9851-4719-8809-d54d98d1dd5b/kube-rbac-proxy/0.log" Sep 11 20:14:04.534670 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.534630 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da9c672e-9851-4719-8809-d54d98d1dd5b/kube-rbac-proxy-metric/0.log" Sep 11 20:14:04.564278 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.564255 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da9c672e-9851-4719-8809-d54d98d1dd5b/prom-label-proxy/0.log" Sep 11 20:14:04.589402 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.589283 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_da9c672e-9851-4719-8809-d54d98d1dd5b/init-config-reloader/0.log" Sep 11 20:14:04.762622 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.762507 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-7xw9h_0b306842-5e77-4670-b2df-41b10871451b/monitoring-plugin/0.log" Sep 11 20:14:04.988960 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:04.988925 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fb8wd_5bce0b0f-6048-4aa3-88a6-233b466228a0/node-exporter/0.log" Sep 11 20:14:05.043199 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:05.042223 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fb8wd_5bce0b0f-6048-4aa3-88a6-233b466228a0/kube-rbac-proxy/0.log" Sep 11 20:14:05.072784 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:05.072730 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fb8wd_5bce0b0f-6048-4aa3-88a6-233b466228a0/init-textfile/0.log" Sep 11 20:14:05.432450 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:05.432355 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d94b4764-2vb6k_8d2faaf3-fabb-4c49-a88f-112010fa4e0f/telemeter-client/0.log" Sep 11 20:14:05.455855 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:05.455821 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d94b4764-2vb6k_8d2faaf3-fabb-4c49-a88f-112010fa4e0f/reload/0.log" Sep 11 20:14:05.481755 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:05.481718 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-55d94b4764-2vb6k_8d2faaf3-fabb-4c49-a88f-112010fa4e0f/kube-rbac-proxy/0.log" Sep 11 20:14:07.040595 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:07.040563 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-94njz_3dbcdeec-40a0-4813-943f-172a623b04c5/networking-console-plugin/0.log" Sep 11 20:14:08.110174 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.110132 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f"] Sep 11 20:14:08.118752 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.118726 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.126270 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.126240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f"] Sep 11 20:14:08.138135 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.136119 2559 scope.go:117] "RemoveContainer" containerID="81954de9bd32254adac0052f76e8b1adb7085ab4e7a0487ce88c87a70743bae6" Sep 11 20:14:08.154684 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.154655 2559 scope.go:117] "RemoveContainer" containerID="ea2bf624e7a27ef5db47f4bec868159ddf612904af5d58ab6207b93aeea39b96" Sep 11 20:14:08.227972 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.227930 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9d68b4776-msw45_b578b307-bf20-4d2a-b0e6-d7659b3645ae/console/0.log" Sep 11 20:14:08.241062 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.241034 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfxxj\" (UniqueName: \"kubernetes.io/projected/da2274ac-406b-410d-b5b5-d767ab2c0b48-kube-api-access-dfxxj\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.241248 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.241204 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-podres\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.241328 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.241306 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-lib-modules\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.241409 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.241372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-sys\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.241486 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.241462 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-proc\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.342231 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.342183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-podres\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.342450 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.342280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-lib-modules\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.342450 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.342316 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-sys\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.342450 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.342352 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-proc\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.342450 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.342408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfxxj\" (UniqueName: \"kubernetes.io/projected/da2274ac-406b-410d-b5b5-d767ab2c0b48-kube-api-access-dfxxj\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.342917 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.342888 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-podres\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.343012 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.342966 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-proc\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.343012 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.342990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-sys\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.343135 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.343044 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/da2274ac-406b-410d-b5b5-d767ab2c0b48-lib-modules\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.350964 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.350930 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfxxj\" (UniqueName: \"kubernetes.io/projected/da2274ac-406b-410d-b5b5-d767ab2c0b48-kube-api-access-dfxxj\") pod \"perf-node-gather-daemonset-p6g2f\" (UID: \"da2274ac-406b-410d-b5b5-d767ab2c0b48\") " pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.439811 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.439723 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:08.741493 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:08.741464 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f"] Sep 11 20:14:08.743042 ip-10-0-137-97 kubenswrapper[2559]: W0911 20:14:08.742998 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda2274ac_406b_410d_b5b5_d767ab2c0b48.slice/crio-838f2a40cedbafbbcf9cec38ee92f710f87cf248437d58d29c07b440faaf414f WatchSource:0}: Error finding container 838f2a40cedbafbbcf9cec38ee92f710f87cf248437d58d29c07b440faaf414f: Status 404 returned error can't find the container with id 838f2a40cedbafbbcf9cec38ee92f710f87cf248437d58d29c07b440faaf414f Sep 11 20:14:09.077405 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:09.077305 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" event={"ID":"da2274ac-406b-410d-b5b5-d767ab2c0b48","Type":"ContainerStarted","Data":"2d5ade6f1b82caeeda994f5596502e3432b64e932dbf7b27d9e2de61d1f6f7b1"} Sep 11 20:14:09.077405 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:09.077389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" event={"ID":"da2274ac-406b-410d-b5b5-d767ab2c0b48","Type":"ContainerStarted","Data":"838f2a40cedbafbbcf9cec38ee92f710f87cf248437d58d29c07b440faaf414f"} Sep 11 20:14:09.078278 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:09.078248 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:09.103749 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:09.103676 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" podStartSLOduration=1.103653046 podStartE2EDuration="1.103653046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 20:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 20:14:09.101238225 +0000 UTC m=+2882.373745428" watchObservedRunningTime="2026-09-11 20:14:09.103653046 +0000 UTC m=+2882.376160248" Sep 11 20:14:10.080304 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:10.080228 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zkfwb_06f12941-963e-4098-aeac-eb39ce1ea795/dns/0.log" Sep 11 20:14:10.099619 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:10.099585 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zkfwb_06f12941-963e-4098-aeac-eb39ce1ea795/kube-rbac-proxy/0.log" Sep 11 20:14:10.160787 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:10.160753 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-48p69_5aed7f34-1818-4df8-95ec-bb9f5ce3c6e2/dns-node-resolver/0.log" Sep 11 20:14:10.812061 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:10.812028 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vqrbw_ac400821-4acd-4ab5-8d9a-03ae365353ec/node-ca/0.log" Sep 11 20:14:12.179207 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:12.179172 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-596df76fbd-kls4w_112e02c3-a598-4433-b801-d748a0a949f9/kube-auth-proxy/0.log" Sep 11 20:14:13.261344 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:13.261307 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qqtnn_447b6ece-d0e6-40cc-811c-faed4cdb4871/serve-healthcheck-canary/0.log" Sep 11 20:14:13.820872 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:13.820839 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rd8p6_11495f40-d751-4bc7-8fc6-ed4d06014372/kube-rbac-proxy/0.log" Sep 11 20:14:13.841949 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:13.841921 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rd8p6_11495f40-d751-4bc7-8fc6-ed4d06014372/exporter/0.log" Sep 11 20:14:13.862413 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:13.862387 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rd8p6_11495f40-d751-4bc7-8fc6-ed4d06014372/extractor/0.log" Sep 11 20:14:16.101765 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:16.101709 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kx729/perf-node-gather-daemonset-p6g2f" Sep 11 20:14:16.253861 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:16.253832 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-88bmh_1e2c2640-d759-41bc-bc41-7f57b39cb976/manager/0.log" Sep 11 20:14:16.582413 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:16.582377 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-88bmh_1e2c2640-d759-41bc-bc41-7f57b39cb976/manager/1.log" Sep 11 20:14:16.825506 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:16.825471 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-kgkdj_d4bffeba-f3e3-49a1-95c6-31b4a31d0755/manager/1.log" Sep 11 20:14:16.855106 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:16.855022 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-kgkdj_d4bffeba-f3e3-49a1-95c6-31b4a31d0755/manager/0.log" Sep 11 20:14:16.871052 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:16.871023 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c47b89ccd-kgkdj_d4bffeba-f3e3-49a1-95c6-31b4a31d0755/copy-manifests/0.log" Sep 11 20:14:19.187670 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:19.187640 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-596df76fbd-kls4w_112e02c3-a598-4433-b801-d748a0a949f9/kube-auth-proxy/0.log" Sep 11 20:14:21.277373 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:21.277336 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-596df76fbd-kls4w_112e02c3-a598-4433-b801-d748a0a949f9/kube-auth-proxy/0.log" Sep 11 20:14:24.906542 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:24.906516 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldhgs_8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7/kube-multus-additional-cni-plugins/0.log" Sep 11 20:14:24.926481 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:24.926457 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldhgs_8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7/egress-router-binary-copy/0.log" Sep 11 20:14:24.945816 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:24.945786 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldhgs_8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7/cni-plugins/0.log" Sep 11 20:14:24.965277 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:24.965246 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldhgs_8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7/bond-cni-plugin/0.log" Sep 11 20:14:24.984684 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:24.984660 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldhgs_8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7/routeoverride-cni/0.log" Sep 11 20:14:25.006973 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:25.006950 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldhgs_8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7/whereabouts-cni-bincopy/0.log" Sep 11 20:14:25.028231 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:25.028205 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-ldhgs_8f889a5c-b68b-4c33-ad71-2aaa6a7a6ad7/whereabouts-cni/0.log" Sep 11 20:14:25.471818 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:25.471776 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ltpvh_f52a92a5-ae1d-4524-9ea7-c44f944a6cea/kube-multus/0.log" Sep 11 20:14:25.497419 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:25.497390 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-crcsn_0b2ec797-aa4b-4aa4-8472-782b11c2e9e4/network-metrics-daemon/0.log" Sep 11 20:14:25.514263 ip-10-0-137-97 kubenswrapper[2559]: I0911 20:14:25.514221 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-crcsn_0b2ec797-aa4b-4aa4-8472-782b11c2e9e4/kube-rbac-proxy/0.log"